Principal Software Engineer, Tech Lead – Databricks Platform
Cardlytics
full-time
Posted on:
Location Type: Remote
Location: United States
Visit company websiteExplore more
Job Level
About the role
- Own day-to-day administration of the Databricks environment on AWS, including workspace configuration, cluster policies, user and group management, access controls, and cost governance.
- Design and evolve the Cardlytics Databricks architecture; establish reference architectures and standards for batch, streaming, and ML workloads.
- Drive adoption of Unity Catalog for centralized RBAC, column-level security, data lineage, and metadata management; implement and maintain security best practices including encryption and audit logging.
- Monitor platform health, define SLAs, and lead performance tuning for Spark jobs, Delta tables, and SQL warehouses; own incident response and root-cause analysis.
- Serve as the primary technical partner for Analysts, ML Engineers, and Business users; develop self-service guides, onboarding materials, and best-practice documentation to reduce friction across all user groups.
- Own the CI/CD strategy for Databricks deployments using Terraform, GitHub Actions, and Databricks Asset Bundles; enforce DevOps practices across the team.
- Lead and mentor a small team of data/platform engineers; provide technical direction, drive engineering excellence, and manage the Databricks vendor relationship.
Requirements
- 7+ years of experience in data engineering or data platform roles, with at least 2 years in a technical lead or architecture capacity.
- Deep, hands-on expertise with the Databricks platform: Delta Lake, Spark (PySpark), Databricks SQL, Unity Catalog, Workflows, and cluster management on AWS.
- Proven experience designing and delivering Lakehouse architectures and large-scale ELT/ETL pipelines in production environments.
- Proficiency in Infrastructure as Code tools, preferably Terraform.
- Demonstrated experience supporting diverse stakeholder groups including Analysts, ML Engineers, and non-technical Business users.
- Strong understanding of data governance, data modeling, and cloud security best practices.
- Excellent communication and collaboration skills, with the ability to translate technical concepts for non-technical audiences.
Benefits
- Flexible paid time off plus company holidays
- Medical, dental, and vision insurance begins on your first day
- 401(k) retirement plan with company match, plan also includes a student loan debt repayment option
- Employee Stock Purchase Plan
- Educational assistance for continuing education
- Lifestyle Spending Account for physical, emotional, and financial wellness
- Complimentary Calm app subscriptions to support employee mental health and wellbeing
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills & Tools
DatabricksDelta LakeSparkPySparkDatabricks SQLUnity CatalogWorkflowsTerraformCI/CDELT/ETL
Soft Skills
communicationcollaborationleadershipmentoringtechnical directionproblem-solvingstakeholder managementdocumentationuser supportengineering excellence