EXL

Databricks Platform Engineer

EXL

full-time

Posted on:

Location Type: Hybrid

Location: Colombia

Visit company website

Explore more

✨ AI Apply
Apply

Tech Stack

AWSCloudPySparkPythonTerraformUnity

About the role

  • Leads the design, build, and optimization of the Databricks ecosystem, including Fivetran ingestion and dbt transformation pipelines.
  • Ensures scalable, reliable compute environments, seamless AWS integration, and automated deployments using Terraform and GitHub Actions.
  • Manage Fivetran connectors and dbt projects for end-to-end data ingestion, transformation, and orchestration across Databricks, S3, and Snowflake.
  • Automate infrastructure provisioning and configuration using Terraform integrated with GitHub Actions.
  • Architect and implement high availability (HA) and disaster recovery (DR) strategies, including cross-region replication and failover.
  • Implement job orchestration, cluster policies, autoscaling, and CI/CD workflows for notebooks, models, and releases.
  • Establish monitoring and alerting for cost, usage, pipeline health, data freshness, and compliance using tools such as Datadog.
  • Collaborate with platform segments and security teams to enforce governance, tagging, rightsizing, SLAs, and audit readiness.
  • Develop and maintain Unity Catalog for governance, lineage, and secure data sharing across platforms.
  • Build and maintain MLflow integrations for model lifecycle management, experimentation tracking, and deployment.

Requirements

  • Graduate in Computer Science, Data Science, or related field.
  • 4-6 years of experience in data engineering or related field.
  • Deep expertise in Databricks administration, cluster management, job scheduling, Unity Catalog, Fivetran, and dbt (Core/Cloud).
  • Proficiency in Terraform, GitHub Actions, and Python/PySpark for automation.
  • Strong knowledge of AWS services, including EKS, ECS, IAM, S3, VPC networking, and CloudWatch.
  • Experience in pipeline reliability engineering, cost optimization, ML CI/CD, HA/DR design, and observability.
Benefits
  • Health insurance
  • 401(k) matching
  • Flexible work hours
  • Paid time off
  • Professional development opportunities
Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools
DatabricksFivetrandbtTerraformGitHub ActionsPythonPySparkAWSMLflowUnity Catalog