Lead Engineer, Data Processing – Databricks
CargoSprint
full-time
Posted on:
Location Type: Remote
Location: United States
Visit company websiteExplore more
Job Level
About the role
- Lead the Data Processing team: technical direction, delivery cadence, coaching, and hiring support (as needed)
- Own core processing systems end-to-end: design, implementation, quality, observability, and on-call readiness
- Drive modernization of legacy processing: simplify flows, reduce failure modes, improve performance and cost
- Build and operate pipelines (batch and/or streaming) with strong data quality, lineage, and backfill strategies
- Establish SLAs/SLOs for key pipelines and improve monitoring, alerting, and incident response
- Partner with application engineering, finance, and analytics to prioritize work and deliver dependable outputs
- Raise the engineering bar via standards, PR reviews, runbooks, and pragmatic architecture decisions
Requirements
- 8+ years building production systems; 2+ years leading engineers or acting as a team tech lead
- Strong experience with data processing systems (ETL/ELT, streaming, batch processing, orchestration)
- Hands-on experience with Databricks and Spark (PySpark/Scala) and strong SQL
- Experience modernizing legacy systems safely and incrementally
- Operational discipline: monitoring, alerting, incident response, and root cause analysis for pipelines
- Ability to drive cross-team alignment and translate requirements into reliable delivery
- Nice to have: Lakehouse patterns (Delta Lake, Iceberg, or Hudi) and data catalog/governance tools
- .Net or Java expertise
- Event-driven architectures (Kafka/Kinesis/PubSub) and streaming pipelines
- FinTech, payments, billing, invoicing, or other high-volume transactional domains
- Experience enabling self-serve analytics for business users
- Logistics, cargo, or supply chain experience
- Spanish language proficiency.
Benefits
- Health and Wellness: Medical, dental, and vision plans for you and your family
- Future-Ready: 401(k) with company match
- Work Life Balance: Generous flexible PTO program and paid holidays
- Grow With Us: Professional development opportunities
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard skills
data processing systemsETLELTstreamingbatch processingorchestrationDatabricksSparkSQL.NetJava
Soft skills
leadershipcoachingcross-team alignmentincident responseroot cause analysis