Lead Data Engineer
The Walt Disney Company
full-time
Posted on:
Location Type: Hybrid
Location: Glendale • California, Washington • 🇺🇸 United States
Visit company websiteSalary
💰 $152,200 - $204,100 per year
Job Level
Senior
Tech Stack
AirflowJavaKafkaPythonSparkSQL
About the role
- Design and build streaming and batch pipelines for AI applications, ensuring low-latency and high-throughput data flows.
- Implement embedding pipelines and vector store integrations to support retrieval, semantic search, and AI assistants.
- Optimize storage and retrieval patterns for scalability, compliance, and cost efficiency.
- Own the design of AI-optimized data stores that align with enterprise data architecture.
- Partner with the AI Core Engineering team to provide reliable pipelines into shared agents, registries, and services.
- Develop reusable data ingestion and transformation frameworks for cross-team adoption.
- Build observability into data pipelines with monitoring, alerting, and dashboards for latency, drift, and failures.
- Implement data quality checks, schema validation, and governance controls.
- Ensure pipelines meet enterprise standards for compliance, audit, and resilience.
- Mentor engineers, conduct design/code reviews, and enforce data engineering best practices.
- Collaborate with infra, product, and governance teams to align on priorities and safe adoption of AI.
- Drive delivery of high-profile AI applications by balancing execution speed with system reliability.
Requirements
- 7+ years of data engineering experience, with at least 2 years in a lead or senior technical role.
- Experience building and scaling streaming data pipelines in large-scale, distributed environments.
- Strong skills in Python, Java and SQL with expert level skill in either Python or Java.
- Proven experience building streaming data pipelines (e.g., Kafka, Flink, Spark, Kinesis).
- Experience with embedding pipelines and vector stores (e.g., Pinecone, Weaviate, FAISS, pgvector).
- Strong knowledge of data modeling, storage optimization, and retrieval patterns for large-scale systems.
- Hands-on experience with workflow orchestration tools (Airflow, Dagster, etc.).
- Strong collaboration and communication skills, able to partner across AI engineering, infra, and product teams.
Benefits
- A bonus and/or long-term incentive units may be provided as part of the compensation package
- Full range of medical, financial, and/or other benefits, dependent on the level and position offered
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard skills
data engineeringstreaming data pipelinesbatch pipelinesdata ingestiondata transformationdata modelingstorage optimizationretrieval patternsdata quality checksschema validation
Soft skills
collaborationcommunicationmentoringdesign reviewscode reviews