Senior Data Engineer – Databricks

SugarCRM

Senior Data Engineer managing Databricks pipelines to drive revenue intelligence for mid-market enterprises. Collaborating with ML and product teams to ensure data reliability and efficiency.

Posted 6/29/2026full-timeRemote • 🇺🇸 United StatesSenior💰 $155,000 - $185,000 per yearWebsite

Tech Stack

Tools & technologies

ApacheAWSAzureERPETLInformaticaPostgresPySparkPythonSparkSQLSSIS

About the role

Key responsibilities & impact

Own Databricks production support for the Sugar Predict data platform, including monitoring, alerting, and incident response across all production data flows
Maintain and report on SLA performance metrics for data pipeline delivery, ensuring visibility into platform health and accountability across internal and external stakeholders
Identify and implement pipeline optimizations that reduce Databricks compute costs, improve throughput, and reduce processing windows while tracking impacts through measurable KPIs
Migrate legacy ETL/ELT pipelines to Databricks, building automation tooling to reduce manual intervention and ensure uninterrupted data delivery during transitions
Support new customers onboarding by provisioning, validating, and hardening tenant data pipelines that deliver reliable, isolated data from day one
Design and build high-performance Databricks pipelines that ingest, transform, and serve ERP and CRM data at scale across both Azure and AWS environments
Own the Delta Lake architecture including schema design, partitioning strategies, data quality enforcement, and incremental processing patterns
Enforce data security best practices across Databricks environments, including role-based access control, secrets management, and compliance requirements for enterprise CRM and ERP data
Implement data quality monitoring and observability across pipeline health and ML model inputs, ensuring data integrity that directly supports Sugar Predict prediction accuracy
Apply and enforce multi-tenant data isolation patterns ensuring reliable, secure data delivery across Sugar Predict enterprise customers
Partner with the Enterprise Architecture team to ensure Sugar Predict data pipelines integrate seamlessly with the broader SugarAI product ecosystem
Support a globally distributed operation through on-call rotation and after-hours incident response, meeting SLAs across multiple time zones
Maintain technical documentation, runbooks, and architectural decision records, contributing to team knowledge sharing and operational readiness across on-call and incident response scenarios
Apply CI/CD best practices to data pipeline development, including version control, automated testing, and deployment tooling to ensure reliable and repeatable pipeline delivery

Requirements

What you’ll need

4+ years of data engineering experience
At least 2 years on Databricks or the Apache Spark ecosystem across Azure and/or AWS
Proficiency in PySpark, SQL, and Python with a strong track record building and operating production-grade pipelines under SLA constraints
Hands-on experience with Delta Lake including schema evolution, ACID transactions, optimize/vacuum lifecycle, and both incremental and streaming processing patterns
Hands-on experience with pipeline performance tuning and compute optimization in production Databricks environments
Solid working knowledge of PostgreSQL including query optimization, schema design, and use as a source or sink in production data pipelines
Experience supporting and maintaining legacy ETL tooling (SSIS, Informatica, custom Python/SQL pipelines, or similar) in production
Experience supporting large-scale multi-tenant architectures with a focus on tenant isolation, per-tenant performance, and data privacy, including navigating tools and platforms that default to single-tenant assumptions
Proven ability to work collaboratively across data science, product, and infrastructure teams, owning end-to-end delivery in a cross-functional environment
Strong understanding of data governance, security, and compliance principles, including access control, data privacy, and protection of sensitive enterprise data across multi-tenant environments

Benefits

Comp & perks

Excellent healthcare package for you and your family
Savings and Investment – 401(k) match
Unlimited Paid Time Off
Paid Parental Leave
Online Legal Services (Rocket Lawyer)
Financial Planning Services (Origin)
Discounted Pet Insurance (Embrace Pet Insurance)
Corporate Benefit Program (Working Advantage). This benefit offers you exclusive travel and entertainment offers and special discounts that are not available to the general public
Health and Wellness Reimbursement Program
Travel Discounts
Educational Resources - Career & Personal Development Program
Employee Referral Bonus Program
We are a merit-based company - many opportunities to learn, excel and grow your career!

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

Data EngineeringSQLPythonPostgreSQLETL/ELT Pipeline MigrationData Quality MonitoringCI/CD Best PracticesPerformance TuningSchema DesignIncident Response

Soft Skills

Collaborative WorkCross-Functional CommunicationProblem Solving