Compass

Senior Scala Data Engineer

Compass

full-time

Posted on:

Location Type: Remote

Location: Brazil

Visit company website

Explore more

✨ AI Apply
Apply

Job Level

Senior

Tech Stack

AirflowApacheAWSETLJavaPythonScalaSpark

About the role

  • Create data pipelines to ingest data using Airflow and Spark;
  • Implement access controls and auditing to protect sensitive data and ensure regulatory compliance;
  • Migrate and integrate existing data into the new platform, ensuring operational continuity and data integrity;
  • Performance tuning: Ability to optimize data pipeline performance by tuning parameters and properly configuring resources;
  • Monitoring and debugging: Ability to monitor and debug pipelines to identify and resolve bottlenecks and performance issues;

Requirements

  • Apache Spark;
  • Apache Airflow;
  • Apache Iceberg;
  • AWS: EMR, Glue, S3, Lake Formation;
  • Data pipelines;
  • Python;
  • Scala or Java;
  • Ability to create and manage data pipelines, scheduling and monitoring ETL tasks;
  • Data governance and security;
  • Monitoring and debugging.
Benefits
  • Health insurance
  • Retirement plans
Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools
Apache SparkApache AirflowApache IcebergAWS EMRAWS GlueAWS S3AWS Lake FormationPythonScalaJava
Soft Skills
performance tuningmonitoringdebugging