Lead Data Engineer

Minor Hotels Europe and Americas

Lead Data Engineer designing and maintaining data pipelines using Databricks and Apache Spark at Capgemini. Collaborating with data scientists on advanced analytics and ensuring data quality and security.

Posted 5/12/2026full-timeRemote • 🇨🇴 ColombiaSeniorWebsite

Tech Stack

Tools & technologies

ApacheAWSAzureCloudETLGoogle Cloud PlatformPySparkPythonSparkSQL

About the role

Key responsibilities & impact

Design, build, and maintain data pipelines and ETL processes using Databricks and Apache Spark.
Optimize data workflows for performance, scalability, and cost efficiency.
Implement data Lakehouse architecture and manage data ingestion from multiple sources.
Collaborate with data scientists and analysts to enable advanced analytics and machine learning workloads.
Ensure data quality, governance, and security across all data assets.
Monitor and troubleshoot Databricks clusters, jobs, and workflows.
Integrate Databricks with cloud services (AWS, Azure, or GCP) and other enterprise systems.
Document processes, standards, and best practices for data engineering.

Requirements

What you’ll need

Hands-on experience with Databricks, Apache Spark, and PySpark.
Strong knowledge of SQL, Python, and data modeling principles.
Experience with cloud platforms (AWS, Azure, or GCP) and their data services.
Familiarity with Delta Lake, Lakehouse architecture, and data governance.
Understanding of CI/CD pipelines and DevOps practices for data workflows.

Benefits

Comp & perks

Health insurance
Professional development opportunities

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

DatabricksApache SparkPySparkSQLPythondata modelingDelta LakeLakehouse architectureCI/CD pipelinesDevOps practices