Director, Sales Operations
Harbor Health
ML/LLM Operations Engineer
Evolent
full-time
Posted on:
Location Type: Remote
Location: Remote • 🇺🇸 United States
Visit company websiteSalary
💰 $120,000 - $140,000 per year
Job Level
JuniorMid-Level
Tech Stack
AWSAzureCloudDockerPythonSQL
About the role
- Ensure our AI systems deliver consistent, reliable, and compliant results in healthcare settings
- Develop and maintain standardized evaluation frameworks to consistently measure LLM performance across relevant healthcare metrics
- Build monitoring systems using Logfire to track AI model performance, detect drift, and alert the team to anomalies
- Create testing infrastructure for prompt versions, model selection, and quality assurance processes
- Design and implement audit sampling processes for continuous quality monitoring and clinical review workflows
- Oversee regulatory compliance processes, including documentation for bias assessments, model cards, and audit trails required in healthcare
- Optimize LLM operations through intelligent model selection, prompt engineering, and cost management strategies
- Support the transition from successful POCs to production-ready services with appropriate testing and validation
- Partner with DevOps on infrastructure requirements while focusing on AI-specific monitoring and optimization
- Create and maintain documentation, runbooks, and operational procedures for all deployed AI systems
- Collaborate with Clinical Support Liaison to incorporate clinical feedback into system improvements
- Prepare regular reports on AI system quality, performance metrics, and compliance status
Requirements
- Bachelor's or master's degree in computer science, data science, or related field
- 2+ years of experience with Python development and at least one production LLM implementation
- Strong proficiency in SQL for complex log analysis and metrics generation
- Demonstrated experience with LLM APIs and frameworks (experience with PydanticAI, LangChain, or similar)
- Experience with monitoring tools and practices for AI systems, including performance metrics, drift detection, and alerting
- Understanding of LLM behavior, prompt engineering, and common failure modes in production
- Experience building evaluation or testing frameworks for AI/ML systems
- Strong communication skills for cross-functional collaboration
- Experience with healthcare AI applications and compliance requirements is preferred
- Familiarity with multiple LLM providers (OpenAI, Anthropic, Google, Azure) is preferred
- Knowledge of Pydantic ecosystem including PydanticAI and Logfire is preferred
- Understanding of LLM evaluation metrics and methodologies is preferred
- Experience building tools for non-technical users is preferred
- Basic knowledge of containerization (Docker) for local testing and development is preferred
- Experience with cloud environments (AWS, Azure) as a user is preferred
- Understanding of API rate limiting, quota management, and cost optimization strategies is preferred
- Knowledge of CI/CD concepts for ML model deployments is preferred
- Experience with regulatory compliance and audit processes is preferred
- Excellent documentation skills and attention to detail is preferred.
Benefits
- Health insurance benefits
- Flexible work arrangements
- Professional development opportunities
- Bonuses
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard skills
PythonSQLLLM implementationLLM APIsPydanticAILangChainmonitoring toolsprompt engineeringevaluation frameworkscontainerization
Soft skills
strong communication skillscross-functional collaborationattention to detail
Certifications
Bachelor's degreeMaster's degree
