AI Quality Lead
Alight Solutions
full-time
Posted on:
Location Type: Hybrid
Location: Lincolnshire • Illinois • United States
Visit company websiteExplore more
Salary
💰 $100,000 - $160,000 per year
Job Level
About the role
- Design and implement the unified AI Quality strategy and SOP, scaling controls by risk tier (Tier 1–3)
- Define and own success metrics, risk tiering, acceptance criteria, and operating thresholds for AI solutions
- Lead the development of golden/adversarial test suites (prompt injection, misuse, safety, bias, functional automation)
- Oversee the creation and maintenance of schema-level UI/API tests, agent contracts, and observability instrumentation
- Own the org-wide evaluation harnesses (e.g., promptfoo, TruLens, Ragas), CI gating, and red-team playbooks
- Manage independent spot checks, factsheets, registry entries, and evidence for governance and audit
- Collaborate with Product Owners, Developers, SRE/Observability, and Governance to ensure compliance and operational excellence
- Drive continuous improvement in fairness, explainability, safety, security, and operational SLOs
- Mentor and grow the AI Quality team, including onboarding and coaching junior engineers
Requirements
- Experience leading or building quality engineering or testing programs (AI/ML experience is a plus, but not required)
- Ability to design and implement test strategies, automation frameworks, and quality standards
- Excellent communication and leadership skills; ability to work cross-functionally and influence stakeholders
- Experience with LLM, RAG, agentic, or classic ML systems
- Familiarity with test automation frameworks, CI/CD pipelines, and evaluation tools (promptfoo, TruLens, Ragas)
- Knowledge of fairness, explainability, safety, and security standards (e.g., OWASP Top 10, MITRE ATLAS, Fairlearn, AIF360)
- Experience with data integrity, lineage, and observability (OpenTelemetry, New Relic, OpenLineage)
- Familiarity with AI infrastructure, orchestration, data integrity, deployments, and model registry practices
- Knowledge of international standards and regulatory frameworks for Responsible AI
- Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science, or related field, or equivalent experience
Benefits
- A variety of health coverage options
- Wellbeing and support programs
- Retirement plans
- Vacation and sick leave
- Maternity, paternity & adoption leave
- Continuing education and training
- Several voluntary benefit options
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard skills
quality engineeringtest strategiesautomation frameworksAI solutionsLLMRAGtest automationdata integrityobservabilitymodel registry
Soft skills
leadershipcommunicationcross-functional collaborationinfluencing stakeholdersmentoringcoaching