Design, develop, and maintain scalable ETL/ELT pipelines using Databricks and Airflow.
Architect and optimize data warehouses and lakehouses in Azure Data Lake and Databricks.
Collaborate with analysts, and business stakeholders to deliver high-quality, reliable datasets.
Work with Healthcare EDI standards (e.g., X12 transactions) to generate flat files or integrate with APIs for Prior Authorization data exchange.
Monitor and troubleshoot pipeline performance, ensuring high availability and reliability.
Drive best practices in data modeling, query optimization, and performance tuning.
Mentor junior engineers and contribute to team knowledge sharing.

Requirements

5+ years of experience in data engineering or related roles.
Strong expertise in Azure cloud services (Azure Data Lake, Synapse Analytics, Azure SQL Database).
Hands-on experience with Databricks (PySpark, Delta Lake, MLflow).
Experience with Python or Scala for data engineering tasks.
Proficiency in Airflow for workflow orchestration.
Advanced SQL skills (query optimization, stored procedures, complex joins).
Solid understanding of data warehousing concepts and distributed computing.
Experience with CI/CD pipelines and DevOps practices for data engineering.
Strong problem-solving skills and ability to work in agile, fast-paced environments.

Benefits

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

ETLELTDatabricksAirflowAzure Data LakePySparkDelta LakeMLflowPythonSQL

Soft Skills

problem-solvingmentoringcollaborationknowledge sharingagilecommunication