Azure Databricks Architect
Minor Hotels Europe and Americas
full-time
Posted on:
Location Type: Office
Location: Atlanta • 🇺🇸 United States
Visit company websiteSalary
💰 $145,000 - $150,000 per year
Job Level
Mid-LevelSenior
Tech Stack
ApacheAWSAzureCloudETLGoogle Cloud PlatformKafkaSparkTerraformUnity
About the role
- Contribute to the design and implementation of scalable data architectures, such as a Lakehouse, using Delta Lake and Unity Catalog.
- Manage and maintain the underlying data infrastructure, which typically exists on a major cloud platform like AWS, Azure, or GCP.
- Implement data governance practices, including data lineage, metadata management, and access controls integrating with other 3rd party products like Immuta,Protegrity for tokenization etc.
- Adhere to software engineering best practices, including participating in code reviews and CI/CD (Continuous Integration/Continuous Deployment) automation.
- Stay up to date with the latest trends and technologies in data engineering and the Databricks ecosystem.
- Design and build robust ETL (Extract, Transform, Load) and ELT workflows to ingest, transform, and load structured and unstructured data from various sources.
- Utilize Databricks features like Delta Live Tables and Databricks Workflows to orchestrate and manage complex data processes.
- Optimize and tune Apache Spark jobs for performance and cost efficiency on large datasets.
- Implement and enforce data security, access control, and compliance policies in Databricks and Azure.
- Hands on experience working on streaming technologies (Kafka, Event Hubs, Kinesis).
- Experience architecting machine learning platforms, advanced analytics workloads and develop and deployment of MLOPS on Databricks.
- Expertise with enterprise security models, networking, and cost governance.
- Should have experience working with DevOps teams to establish deployment practices using Terraform or similar.
- Optimize Databricks performance (auto-scaling, caching, delta optimization, job tuning, cost optimization).
- Prior experience in leading enterprise Azure Databricks implementations
Requirements
- Contribute to the design and implementation of scalable data architectures, such as a Lakehouse, using Delta Lake and Unity Catalog.
- Manage and maintain the underlying data infrastructure, which typically exists on a major cloud platform like AWS, Azure, or GCP.
- Implement data governance practices, including data lineage, metadata management, and access controls integrating with other 3rd party products like Immuta,Protegrity for tokenization etc.
- Adhere to software engineering best practices, including participating in code reviews and CI/CD (Continuous Integration/Continuous Deployment) automation.
- Stay up to date with the latest trends and technologies in data engineering and the Databricks ecosystem.
- Design and build robust ETL (Extract, Transform, Load) and ELT workflows to ingest, transform, and load structured and unstructured data from various sources.
- Utilize Databricks features like Delta Live Tables and Databricks Workflows to orchestrate and manage complex data processes.
- Optimize and tune Apache Spark jobs for performance and cost efficiency on large datasets.
- Implement and enforce data security, access control, and compliance policies in Databricks and Azure.
- Hands on experience working on streaming technologies (Kafka, Event Hubs, Kinesis).
- Experience architecting machine learning platforms, advanced analytics workloads and develop and deployment of MLOPS on Databricks.
- Expertise with enterprise security models, networking, and cost governance.
- Should have experience working with DevOps teams to establish deployment practices using Terraform or similar.
- Optimize Databricks performance (auto-scaling, caching, delta optimization, job tuning, cost optimization).
- Prior experience in leading enterprise Azure Databricks implementations
Benefits
- Flexible work
- Healthcare including dental, vision, mental health, and well-being programs
- Financial well-being programs such as 401(k) and Employee Share Ownership Plan
- Paid time off and paid holidays
- Paid parental leave
- Family building benefits like adoption assistance, surrogacy, and cryopreservation
- Social well-being benefits like subsidized back-up child/elder care and tutoring
- Mentoring, coaching and learning programs
- Employee Resource Groups
- Disaster Relief
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard skills
Delta LakeUnity CatalogETLELTApache SparkMLOPSTerraformdata governancedata lineagemetadata management
Soft skills
collaborationleadershipcommunicationproblem-solvingadaptabilityattention to detailcritical thinkingtime managementteamworkanalytical thinking