Senior Data Engineer
Wpromote
AirflowAmazon RedshiftAWSAzureBigQueryCloudPythonSQL
Senior Data Engineer
EXL
full-time
Posted on:
Location Type: Remote
Location: Remote • 🇺🇸 United States
Visit company websiteJob Level
Senior
Tech Stack
AzureETLPySparkPythonSparkSQLVault
About the role
- Lead the design, development, and deployment of scalable data pipelines and workflows using core Azure data services such as Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage (ADLS), along with Azure Databricks.
- Architect and optimize data solutions leveraging **Medallion Architecture on Delta Lake using Azure Databricks** as part of the overall Azure data platform strategy.
- Develop and optimize ETL/ELT processes with Azure Data Factory and Azure Databricks to ensure efficient data ingestion, transformation, and delivery.
- Collaborate closely with data scientists, analysts, and business stakeholders to translate business needs into effective Azure-based data engineering solutions.
- Lead and mentor a team of data engineers, enforcing best practices in coding, testing, deployment, and documentation within the Azure environment.
- Ensure data governance, quality, security, and compliance standards are maintained across Azure data pipelines.
- Implement secure authentication and authorization mechanisms using **Azure Active Directory (AAD)** and manage secrets and keys securely using **Azure Key Vault**.
- Implement CI/CD pipelines for automated deployment and monitoring using Azure DevOps.
- Stay updated with Azure data services innovations and incorporate new technologies to enhance data infrastructure and team productivity.
Requirements
- 8+ years of data engineering experience with a strong background working on the Azure data platform.
- Solid hands-on experience with **Azure Databricks**, including Spark, PySpark, Delta Lake, and implementing Medallion Architecture.
- Hands-on experience with Azure Data Factory for building scalable ETL/ELT pipelines.
- Practical experience working with Azure Synapse Analytics and Azure Data Lake Storage (ADLS).
- Strong skills in Python, SQL( T-SQL) & PowerShell.
- Experience with Azure Active Directory (AAD) integration for authentication and role-based access control.
- Experience managing secrets and certificates with Azure Key Vault.
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard skills
data engineeringETLELTAzure DatabricksSparkPySparkDelta LakeAzure Data FactoryPythonSQL
Soft skills
leadershipmentoringcollaborationcommunication
