Design, build, and optimize high-throughput ETL pipelines using PySpark and cloud services to manage the flow of multimodal AV sensor logs
Collaborate directly with ML Engineers to productionize, scale, and performance-tune the model inference pipelines, focusing on maximizing data throughput and minimizing operational costs
Implement robust data quality checks, schema validation, and monitoring on all raw input data and on the structured, searchable metadata
Identify bottlenecks in data movement and processing, improve the speed and efficiency of data preparation, and downstream data retrieval for dashboards and data search functionalities.
Serve as the liaison between the Data Science teams and the Data Platform team, advocating for and implementing infrastructure improvements necessary for long-term scalability and reliability.
Requirements
5+ years of professional experience in a Data Engineering role, specifically focused on supporting complex analytics initiatives and machine learning.
Bachelor's or Master's degree in Computer Science, Engineering, or a related field.
Expert proficiency in PySpark and distributed computing frameworks for processing petabyte-scale datasets.
Deep working knowledge of cloud ecosystems (AWS, GCP, or Azure) and modern data lake/warehouse technologies.
Demonstrated experience in migrating prototype data pipelines/scripts into fully managed, production-ready, fault-tolerant ETL systems.
Benefits
Comprehensive healthcare suite including medical, dental, vision, life, and disability plans. Domestic partners who have been residing together at least one year are also eligible to participate.
Health Savings and Flexible Spending Healthcare and Dependent Care Accounts available.
Rich retirement benefits, including an immediately vested employer safe harbor match.
Generous paid parental leave as well as a phased return to work.
Flexible vacation policy in addition to paid company holidays.
Total Wellness Program providing numerous resources for overall wellbeing
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.