Principal Software Engineer – AI Platform, Production Engineering, Reliability

CVS Health

Principal Software Engineer leading production excellence for AI Platform at CVS Health. Drive operational readiness and observability for AI services with high availability and performance standards.

Posted 5/28/2026full-timeRemote • Texas • 🇺🇸 United StatesLead💰 $144,200 - $288,400 per yearWebsite

Tech Stack

Tools & technologies

AWSAzureCloudDistributed SystemsGoogle Cloud Platform

About the role

Key responsibilities & impact

Own and evolve production operations strategy for AI/ML platforms and services
Define SLOs, SLIs, and error budgets for AI systems
Lead root cause analysis (RCA) and drive systemic improvements post-incident
Establish operational readiness standards for launching new AI capabilities
Build frameworks for on-call excellence , incident response, and escalation
Design and implement end-to-end observability systems across AI workloads
Implement model observability (drift detection, data skew, performance degradation)
Build internal platforms and tooling for automated incident detection and response
Mentor senior engineers and influence cross-team architectural decisions

Requirements

What you’ll need

10+ years in software engineering, production engineering, or SRE roles
Deep experience operating large-scale distributed systems in production
Proven track record building monitoring, observability, and alerting systems
Strong expertise in incident management and production support models
Experience working with cloud platforms (Azure, AWS, GCP)

Benefits

Comp & perks

Medical, dental, and vision coverage
Paid time off
Retirement savings options
Wellness programs
Other resources based on eligibility

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

AI platformsML platformsSLOsSLIsroot cause analysisobservability systemsmodel observabilityincident detectionautomated responsedistributed systems

Soft Skills

mentoringinfluencingleadership