Staff Machine Learning Engineer

Toast

Staff Machine Learning Engineer at Toast responsible for leading ML platform architecture and collaboration across engineering teams. Driving core infrastructure for ML capabilities.

Posted 7/9/2026full-timeRemote • California, Florida, Illinois, Missouri, New York, North Carolina, Ohio, Pennsylvania, Texas • 🇺🇸 United StatesLead💰 $151,000 - $309,000 per yearWebsite

Tech Stack

Tools & technologies

Distributed SystemsGoJavaKotlinPythonScala

About the role

Key responsibilities & impact

Own technical direction of the ML Platform — feature store, model hosting and serving, experimentation, training infrastructure — driving architectural decisions around scalability, reliability, latency, and cost
Lead design and delivery of large-scope platform initiatives from conception through production, coordinating across ML, data, and infrastructure teams
Identify and resolve systemic technical challenges: online/offline feature parity, model deployment friction, experimentation velocity, GPU utilization, cross-team dependencies
Set and maintain a high engineering quality bar through hands-on code contributions, design reviews, and mentorship of platform and ML-adjacent engineers
Partner with ML engineering, data science, product, and platform leadership to translate ML strategy into technical roadmaps
Define the paved paths ML teams use to ship models safely — from feature registration through canary rollout, monitoring, and rollback
Leverage AI-augmented development tools to increase development velocity and code quality

Requirements

What you’ll need

8+ years delivering complex backend or infrastructure systems at scale
Direct experience building or operating core ML infrastructure — feature stores, model serving, experimentation platforms, training orchestration, or equivalent
Mastery of a modern backend language such as Python, Java, Kotlin, Go, or Scala
Deep proficiency with distributed systems concepts: consistency, latency, throughput, fault tolerance, and observability
Strong understanding of data modeling, query languages, and the online/offline data patterns that underpin ML systems
Demonstrated technical leadership, with ability to drive cross-team alignment and influence engineering, product, and business stakeholders
Bachelor's degree in Computer Science or a related field, or equivalent practical experience.

Benefits

Comp & perks

Competitive compensation and benefits programs
Flexibility to meet Toasters’ changing needs
Health insurance
401(k) matching

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

PythonJavaKotlinGoScalaDistributed SystemsData ModelingQuery LanguagesFeature StoresModel Serving

Soft Skills

Cross-Team AlignmentInfluencing StakeholdersMentorship

Certifications

Bachelor's Degree in Computer Science