Design and implement batch and streaming data pipelines on Azure Databricks (PySpark); author Spark SQL for transformations and analytics.
Orchestrate workflows with Azure Data Factory and/or Synapse pipelines; integrate with Lake Storage.
Model and maintain lakehouse structures (e.g., Delta Lake), ensuring robust partitioning, schema evolution, and performance.
Implement data quality checks, observability, and SLAs across pipelines.
Optimize jobs for cost and performance (cluster sizing, caching, shuffle reduction, partition strategy).
Collaborate with architecture/platform teams on CI/CD (Azure DevOps), secrets management (Key Vault), and security (RBAC, PIM).
Contribute to governance and metadata practices; document lineage and technical design.
Support release cycles, incident triage, and production hardening; drive continuous improvements.

Requirements

Hands on expertise in PySpark (DataFrames, Spark SQL, performance tuning) on Azure Databricks-this is the primary skill focus.
Strong SQL and data modeling for analytical workloads (star/snowflake, lakehouse patterns).
Proven delivery with Azure Data Factory/Synapse for pipeline orchestration and scheduling.
Solid knowledge of Azure storage (ADLS Gen2, partitions, file formats-Parquet/Delta).
Version control and CI/CD with Git/Azure DevOps; automated testing in data pipelines.
Experience operating pipelines in production (monitoring, alerting, reliability).
Nice to have (preferred) Microsoft Fabric exposure is nice to have (not mandatory)
Data governance tools (e.g., Purview), Power BI integration, Delta Live Tables.
Python packaging best practices; basic PowerShell for automation.
Domain experience in financial services/asset management.

Benefits

Flexible work
Healthcare including dental, vision, mental health, and well-being programs
Financial well-being programs such as 401(k) and Employee Share Ownership Plan
Paid time off and paid holidays
Paid parental leave
Family building benefits like adoption assistance, surrogacy, and cryopreservation
Social well-being benefits like subsidized back-up child/elder care and tutoring
Mentoring, coaching and learning programs
Employee Resource Groups
Disaster Relief

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

PySparkSpark SQLdata modelingSQLAzure Data FactoryAzure SynapseDelta Lakedata quality checksperformance tuningPython

Soft Skills

collaborationincident triagecontinuous improvement