Compass

AWS Data Engineer – Mid

Compass

full-time

Posted on:

Location Type: Remote

Location: Brazil

Visit company website

Explore more

✨ AI Apply
Apply

Tech Stack

AirflowApacheAWSAzureCloudEC2GrafanaPython

About the role

  • Contribute to a data platform migration and modernization project;
  • Map, analyze, and understand existing notebooks and jobs in Databricks (Azure);
  • Define and execute migration waves for workloads to AWS EMR (EC2 and/or Serverless);
  • Implement and maintain orchestration of pipelines and jobs using Apache Airflow (MWAA);
  • Build and maintain CI/CD pipelines for deployment, versioning, and automation;
  • Structure and organize the data catalog, metadata, and access governance using AWS Lake Formation;
  • Ensure pipeline observability by integrating logs and metrics with Grafana;
  • Structure data in Landing, Silver, and Gold layers using Hudi and Iceberg, ensuring adherence to the current model in ADLS;
  • Participate in acceptance testing, change management processes, and production deployments;
  • Promote best practices and conduct knowledge transfer to the involved teams.

Requirements

  • Experience with Databricks (Azure), including mapping, analyzing, and refactoring notebooks and jobs;
  • Experience with AWS cloud services (EMR, Glue, Lambda);
  • Experience with data pipeline orchestration using Apache Airflow (MWAA);
  • Experience structuring data in layered architectures (Landing, Silver, and Gold);
  • Experience with data governance, including catalog, metadata, and access control via AWS Lake Formation;
  • Experience with pipeline observability, using logs, metrics, and integration with Grafana;
  • Experience with Python.
Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools
DatabricksAWS EMRApache AirflowCI/CD pipelinesAWS Lake FormationGrafanaHudiIcebergPythondata pipeline orchestration
Soft Skills
knowledge transferchange managementacceptance testingbest practices