Design and operate large-scale data pipelines that generate training datasets used for machine learning training and experimentation
Develop infrastructure that supports distributed training workflows using technologies such as Pytorch, Ray Data, and Ray Train, etc.
Integrate ML pipelines with workflow orchestration systems (e.g., Flyte, Airflow, or similar) to enable reliable multi-stage training workflows
Improve reproducibility and observability of ML pipelines through dataset validation, monitoring, and automated testing
Optimize performance and resource utilization across distributed compute systems used for data processing and model training
Partner closely with ML engineers to enable efficient large-scale experimentation and model iteration
Lead architectural improvements to ensure our offline ML pipelines remain scalable, reliable, and cost-efficient

Requirements

Strong experience building large-scale ML pipelines
Experience working with distributed computing frameworks such as Ray,
Spark, Flink and familiarity in the Ray ecosystem (Ray Data, Ray Train) for distributed data processing and model training
Experience building infrastructure for training data generation, dataset preparation, or ML feature pipelines
Deep experience designing and operating production-grade data pipelines
Strong programming skills in Python and experience working with large-scale distributed workloads
Experience with modern data infrastructure (data lakes, warehouses, orchestration systems, streaming platforms)
Strong systems thinking, with the ability to reason about performance, scalability, reliability, and cost tradeoffs in distributed systems
Proven ability to lead technical direction and influence architectural decisions across teams without formal authority

Benefits

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

machine learning pipelinesdistributed computing frameworksRaySparkFlinkPythondata lakesdata warehousesworkflow orchestrationdataset validation

Soft Skills

systems thinkingleadershipinfluencecollaborationproblem-solving