AI Quality Assurance Analyst

OneDigital

AI Quality Assurance Analyst responsible for ensuring AI applications' reliability and accuracy for OneDigital. Involves testing, evaluation, and collaboration with product teams.

Posted 7/2/2026full-timeRemote • 🇺🇸 United StatesMid-LevelSenior💰 $100,000 - $110,000 per yearWebsite

About the role

Key responsibilities & impact

Design and execute test plans for AI-powered applications across both output quality and functional behavior
Evaluate AI responses for accuracy, consistency, hallucination, and edge case handling
Test application workflows, UI behavior, and integrations across internal and client-facing tools
Identify, document, and track defects using standard issue tracking tools (e.g., Jira or equivalent)
Partner with AI Builders and Product Managers to define acceptance criteria and quality standards for AI features
Develop and maintain repeatable test cases and QA documentation as AI applications evolve
Conduct regression testing as applications are updated or new features are released
Collaborate with the Principal Engineer and product team to surface quality risks early in the build process
Contribute to defining what “good” looks like for AI output quality at OneDigital, helping shape internal QA standards for LLM-based applications
Support testing across multiple business lines including Employee Benefits, Retirement, Wealth, and P&C

Requirements

What you’ll need

Experience in a QA, testing, or quality-focused role, with some exposure to AI or LLM based products preferred
Strong ability to evaluate AI-generated outputs for quality, accuracy, and appropriateness
Familiarity with prompt-driven systems and an intuition for how language models can fail or produce unexpected results
Experience writing and executing test cases and documenting defects clearly
Comfort working in no-code and low-code environments
Strong written communication skills, with the ability to describe issues precisely and concisely
Curiosity and adaptability in a fast-moving, emerging technology space
Ability to work independently and manage multiple testing workstreams at once

Benefits

Comp & perks

Health, wellbeing, retirement, and other financial benefits
Paid time off
Overtime pay for non-exempt employees
Robust learning and development programs
Reimbursement of job-related expenses per company policy
Employee perks and discounts

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

Test Case WritingQuality AssuranceAI Application TestingFunctional TestingIssue TrackingAcceptance Criteria DefinitionQA DocumentationWorkflow TestingIntegration TestingEdge Case Handling

Soft Skills

Strong Written CommunicationCuriosityAdaptabilityIndependent WorkMulti-Tasking