Be part of a team supporting the development of an AI/ML platform that will leverage generative AI to relieve user burden across the Centers for Medicare and Medicaid Services (CMS)
Design and maintain robust, scalable automated test frameworks for web apps, APIs, and AI-driven features
Develop automated pipelines for model validation, bias detection, and drift monitoring in ML-based systems
Define and execute test plans that include functional, regression, performance, and AI-specific testing strategies
Partner with AI/ML engineers to create test datasets and simulate real-world user behavior and edge cases
Implement CI/CD-integrated quality gates to ensure AI releases meet defined reliability and fairness metrics
Mentor junior engineers and help establish best practices in intelligent automation and AI testing
Advocate for ethical AI standards and contribute to cross-functional discussions about responsible AI
Requirements
3-5 years of experience in software QA/automation engineering, with at least 1 year focused on AI/ML product testing
Strong proficiency in at least one automation framework (e.g., Selenium, Cypress, Playwright) and scripting language (Python preferred)
Hands-on experience with testing AI models, including validation metrics, dataset versioning, and model explainability
Familiarity with tools like MLflow, TensorBoard, or similar for AI observability
Understanding of model bias, fairness audits, and compliance considerations in government or regulated environments
Experience working in Agile DevSecOps environments with modern CI/CD tooling (Jenkins, GitHub Actions, GitLab, etc.)
Strong collaboration skills with clear communication across engineering, data science, and product teams
Ability to obtain and maintain a Public Trust security clearance
Offers of employment are contingent upon acceptable results of a background investigation
Benefits
Medical, dental, vision insurance (fully paid for employees)
15 days of paid leave
7 days of sick leave
2 days bereavement leave
11 paid Federal holidays
Up to 40 hours for jury duty
401K with 4% employer contribution (and no vesting period)
Up to 4 weeks of paid paternity and maternity leave
Company provided laptop
$5,000 per year for professional development
$600 per year for technical supplies and equipment
$2,000 referral bonus
Life and disability insurance
HSA and FSA
Legal Shield and ID Shield Voluntary Benefits
Flexible working hours and remote opportunities to help maintain a healthy work-life balance
Opportunity to work in a collaborative, motivated team focused on modernizing government services with cutting-edge technology
ATS Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard skills
software QAautomation engineeringAI/ML product testingautomation frameworkscripting languagetesting AI modelsvalidation metricsdataset versioningmodel explainabilityautomated test frameworks
Soft skills
collaborationcommunicationmentoringadvocacy for ethical AIbest practices establishment