AI Quality & Evaluation Lead

Alight Solutions

AI Quality & Evaluation Lead enabling responsible AI adoption at Alight. Collaborating with AI engineering, Data Scientists, and Product teams to ensure accuracy and compliance.

Posted 7/24/2026full-timeRemote • Illinois • 🇺🇸 United StatesSenior💰 $150,000 - $200,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in AI Quality Assurance by defining evaluation frameworks, establishing governance standards, and developing metrics for Generative AI performance. Proficient in translating complex trust concepts into actionable technical controls while ensuring compliance with regulatory requirements.

Highest-signal resume keywords

Expert-Level PythonEvaluation FrameworksData Science ExperienceAI Quality AssuranceGovernance Standards

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

PythonPandasScikit-learnRAGASTruLensMLflowGenerative AI MetricsStatistical AnalysisBias EvaluationModel Calibration

Soft Skills

CollaborationCommunicationProblem-Solving

Industry Keywords

AI GovernanceData ScienceMachine Learning EngineeringQuality AssuranceRisk Management

Tech Stack

Tools & technologies

PandasPythonScikit-Learn

About the role

Key responsibilities & impact

Partnering directly with AI Engineers, Application Developers and Data Scientists during the design phase to define technical quality acceptance criteria and fit-for-use requirements
Embedding quality considerations into model and system architecture from the onset, specifically for complex patterns like RAG and autonomous Agents
Defining golden truth requirements and evaluation dataset standards; partner with Data Science teams to ensure datasets reflect production-level complexity
Defining quality and evaluation expectations for third-party AI systems and vendor-supplied models, ensuring consistent governance standards regardless of model origin
Designing and maintaining structured evaluation framework that assesses AI system against defined quality bars (e.g. Goodness-of-Fit, Calibration, Stability)
Developing automated metrics for Generative AI performance including Groundedness (Hallucination detection), Faithfulness, Completeness, and other domain-relevant metrics
Defining and operationalize fairness and bias evaluation criteria, including demographic parity assessments and disparate impact testing for client-facing AI systems
Calibrating evaluation thresholds and monitoring cadence to AI risk tier, ensuring proportionate controls without over-engineering lower-risk use cases
Identifying and document technical AI governance controls that enable automated compliance with performance and risk obligations
Establishing drift and ongoing monitoring requirements, defining statistical triggers for feature and concept drift that necessitate model intervention
Providing objective, data-driven evaluation outputs that support AI governance reviews and risk classification
Maintaining authoritative documentation of AI controls to support audit, regulatory review, and internal assurance activities

Requirements

What you’ll need

5–8+ years of experience in Data Science, ML Engineering, or AI Quality
Practical experience partnering with engineers to design RAG, LLM-based Agents, or traditional ML pipelines
Expert-level Python (Pandas, Scikit-learn)
Experience with evaluation frameworks (e.g., RAGAS, TruLens, or MLflow)
Demonstrated ability to translate abstract trust concepts into mathematical metrics and enforceable technical controls
Ability to bridge the gap between high-level governance policy and low-level code implementation
Bachelor’s degree in a technical field (e.g., Computer Science, Computer Systems Design) or equivalent professional experience

Benefits

Comp & perks

health, dental and vision coverages starting Day One
wellbeing programs
retirement plans with contribution matching
generous time off
parental leave
continuing education
career growth opportunities
flexible working arrangements