FREE ACCESS
5,000–10,000 jobs/day
See all jobs on JobTailor
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Tech Stack
Tools & technologiesAirflowAWSAzureCloudETLGrafanaKafkaPySparkPythonSparkSplunkSQLTerraformUnityVault
About the role
Key responsibilities & impact- Ingestion & Transformation: Design and optimize high-volume ETL/ELT pipelines using Delta Live Tables (DLT) and PySpark, ensuring data integrity across the Bronze, Silver, and Gold layers.
- Workflow Orchestration: Develop and maintain sophisticated pipelines using Databricks Workflows or Airflow, focusing on modularity, reusability, and automated error handling.
- Streaming & Real-time Integration: Implement real-time data flows utilizing Structured Streaming and Kafka/Event Hubs to enable immediate data availability for downstream consumption.
- Data Security & Privacy: Enforce data anonymization and fine-grained access controls to ensure compliance with global regulations (GDPR/CCPA/HIPAA).
- DataOps & DevOps: Implement CI/CD patterns using Databricks Asset Bundles (DABs), Terraform, and Git to automate environment parity and deployments.
- Open Table Formats: Manage and optimize Delta Lake storage, utilizing advanced features like Liquid Clustering, Z-Ordering, and Change Data Feed (CDF).
- Compute Engine Optimization: Drive cost efficiency and performance by optimizing Spark configurations, Photon engine utilization, and Serverless SQL Warehouses.
- Observability & Monitoring: Integrate comprehensive monitoring and alerting (e.g., Databricks System Tables, Grafana, or Splunk) to rapidly identify bottlenecks and troubleshoot production issues.
Requirements
What you’ll need- 6+ Years of hands-on, progressive experience in Data Engineering, with at least 5 years focused heavily on the Databricks platform.
- Architectural Understanding: Expert knowledge of Medallion Architecture, Data Vault 2.0 or Dimensional Modeling, and modern Lakehouse design patterns.
- Scale Expertise: Proven track record of building and managing large-scale data infrastructure (Petabyte-scale) in cloud-native environments.
- Industry Experience: Experience in the Insurance or Financial Services industry is preferred (focusing on claims, policy, or risk data).
- Technical Toolset:
- Cloud Environment: Azure (preferred), AWS.
- Databricks Stack: Unity Catalog, Delta Live Tables, Databricks SQL, MLflow.
- Core Languages: Expert-level SQL, Python, and PySpark.
- Supporting Tools: dbt (Databricks adapter), Git, and Orchestration tools.
Benefits
Comp & perks- Work From Home 📊 Check your resume score for this job Improve your chances of getting an interview by checking your resume score before you apply. Check Resume Score
ATS Keywords
✓ Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills & Tools
ETLELTPySparkDatabricks WorkflowsStructured StreamingKafkaCI/CDTerraformSQLData Vault 2.0