Senior Data Engineer – Azure
Dataside
AzureSparkUnity
Data Engineer – Specialist
Compass
full-time
Posted on:
Location Type: Remote
Location: Remote • 🇧🇷 Brazil
Visit company websiteJob Level
Mid-LevelSenior
Tech Stack
AirflowApacheAWSCloudETLPythonScalaSpark
About the role
- Build data pipelines to ingest data using Airflow and Spark;
- Implement access controls and audit mechanisms to protect sensitive data and ensure compliance with regulations;
- Migrate and integrate existing data into the new platform, ensuring continuity of operations and data integrity;
- Optimize data pipeline performance by tuning parameters and correctly configuring resources;
- Monitor and debug pipelines to identify and resolve bottlenecks and performance issues.
Requirements
- Apache Spark, Apache AirFlow, Apache Iceberg;
- AWS Cloud: Glue, S3, EMR, Lake Formation, Lambda;
- Python and Scala;
- Design and manage data pipelines, including scheduling and monitoring ETL jobs.
Benefits
- Don’t meet all the requirements for the role?
- That’s okay! At Compass UOL, we encourage continuous development of new talent and turn challenges into opportunities.
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard skills
Apache SparkApache AirFlowApache IcebergAWS GlueAWS S3AWS EMRAWS Lake FormationAWS LambdaPythonScala
