Senior Manager, AI Performance & Operations

Devoted Health

Senior Manager overseeing AI performance and operations for a healthcare company focused on improving the well-being of older Americans. Managing high-risk AI workflows and ensuring operational outcomes.

Posted 5/22/2026full-timeRemote • 🇺🇸 United StatesSenior💰 $110,000 - $172,000 per yearWebsite

About the role

Key responsibilities & impact

Own the health of major voice and text AI workflows, ensuring business, quality, safety, and operational outcomes are met.
Maintain a current view of all major workflows, including status, maturity, failure modes, and next actions.
Monitor weekly indicators (containment, escalation, complaints) and oversee production dashboards to track behavior independently.
Lead response for AI regressions; distinguish between model, prompt, or orchestration issues and drive post-mortems to closure.
Run structured cycles (hypothesis, testing, rollout/rollback) for 'mini-launches' like model swaps, prompt changes, and tool updates.
Design eval systems using golden datasets, LLM-as-judge workflows, and human review queues for production and pre-production.
Guide new workflows during incubation to ensure they launch with proper instrumentation, eval coverage, and safety models.
Translate technical changes into plain language for leadership, explaining what broke, why it matters, and who is fixing it.

Requirements

What you’ll need

Strong production experience with LLMs, RAG, voice AI, agentic orchestration, or equivalent ML systems.
Demonstrated experience designing or operating rubrics, scoring frameworks, regression tests, and production-sampling workflows.
A clear mental model for distinguishing failure modes, such as model reasoning, tool failure, retrieval issues, and workflow logic.
Comfort with active experimentation—including prompt/context tuning, retrieval redesign, and safe model migrations—beyond simple issue triage.
Excellent cross-functional judgment and the ability to communicate clearly in ambiguous, high-accountability environments.
Deep personal accountability with a drive to own results rather than just producing recommendations.
Experience in healthcare, regulated workflows, contact center operations, or high-stakes service quality/reliability functions.

Benefits

Comp & perks

Employer sponsored health, dental and vision plan with low or no premium
Generous paid time off
$100 monthly mobile or internet stipend
Stock options for all employees
Bonus eligibility for all roles excluding Director and above; Commission eligibility for Sales roles
Parental leave program
401K program
And more....

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

LLMsRAGvoice AIagentic orchestrationrubricsscoring frameworksregression testsproduction-sampling workflowsprompt tuningretrieval redesign

Soft Skills

cross-functional judgmentclear communicationpersonal accountabilitydrive to own results