Senior Quality Assurance Engineer

Pearson VUE

Senior Quality Assurance Engineer focusing on testing AI-powered support tools. Ensuring quality and accuracy of AI responses across complex integrations in EdTech platforms.

Posted 7/20/2026full-timeColombo • 🇱🇰 Sri LankaSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in software quality engineering with a focus on testing AI/ML systems, conversational AI, and LLM-based applications. Proficient in building evaluation datasets, automated testing frameworks, and validating cross-platform integrations within EdTech environments.

Highest-signal resume keywords

Software Quality EngineeringAI/ML Systems TestingLLM-Based Application TestingAPI Testing (REST)Test Automation Frameworks (Selenium, Playwright, Cypress)

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Prompt EvaluationResponse GradingFactual Accuracy MeasurementRegression TestingRetrieval-Augmented Generation (RAG)Semantic Search TestingVector-Based Retrieval SystemsScripting (Python, JavaScript, Java)Performance/Load TestingAccessibility Testing (WCAG 2.1 AA)

Tools & Technologies

SeleniumPlaywrightCypressJiraConfluence

Industry Keywords

Conversational AIChatbotsNLP-Driven ProductsLMS PlatformsEdTech

Tech Stack

Tools & technologies

CypressDistributed SystemsJavaJavaScriptJMeterPythonSelenium

About the role

Key responsibilities & impact

Design and execute test strategies for LLM-powered conversational flows, including response accuracy, relevance, assumption detection, and hallucination prevention.
Build and maintain evaluation datasets to measure assistant performance against target metrics (95% accuracy, 75% CSAT).
Validate context injection — ensure the assistant correctly receives and uses runtime context (role, product, LMS type, browser, workflow state, structured error/diagnostic codes) to tailor responses.
Test the context-sharing contract between front-end applications and the AI Support Assistant — verifying session storage writes, schema compliance, and read interfaces.
Validate cross-platform behavior across major LMS integrations (e.g., Canvas, Blackboard, Moodle, D2L) and multiple product lines across the courseware portfolio.
Test entitlement and enrollment and third-party content-provider provisioning flows.
Build automated regression suites for assistant response quality, context propagation, and UI behavior (chat widget placement, discoverability, mobile responsiveness).
Develop performance and load testing strategies for projected scale (millions of interactions annually, growing year over year).
Establish monitoring and alerting for production assistant accuracy, escalation rates, and context-pass-through failures.
Partner with ML/AI engineers to define acceptance criteria for model updates and prompt changes.
Work closely with product management to translate user research findings (e.g., accuracy as the #1 trust driver, step-based answers over article links) into testable requirements.
Coordinate with backend platform and Customer Success teams on dependency validation.

Requirements

What you’ll need

5+ years in software quality engineering, with at least 2 years testing AI/ML systems, conversational AI, chatbots, or NLP-driven products.
Hands-on experience testing LLM-based applications — prompt evaluation, response grading, factual accuracy measurement, and regression testing for non-deterministic outputs.
Hands-on experience validating Retrieval-Augmented Generation (RAG) pipelines, including retrieval accuracy, context grounding, and relevance validation.
Experience testing semantic search and vector-based retrieval systems (embeddings, similarity scoring, ranking relevance).
Experience with LLM Evaluator frameworks and LLM-as-judge methodologies for automated, scalable scoring of model outputs.
Experience building evaluation harnesses for LLM outputs, including automated scoring and human-in-the-loop review pipelines.
Strong experience with API testing (REST) and integration testing across distributed systems.
Proficiency with test automation frameworks (Selenium, Playwright, Cypress, or similar) and CI/CD pipelines.
Strong scripting/programming ability (e.g., Python, JavaScript, or Java) to build custom test tooling, evaluation harnesses, and data pipelines.
Experience testing across multiple browsers, devices, and platforms — including mobile web.
Solid understanding of session/local storage, client-side state management, and front-end data contracts.
Familiarity with e-commerce or EdTech platforms — checkout flows, entitlements, user provisioning.
Preferred: Experience with LMS platforms (e.g., Canvas, Blackboard, Moodle) and LTI integrations.
Preferred: Familiarity with MCP (Model Context Protocol) or similar AI tool-integration patterns.
Preferred: Knowledge of accessibility testing (WCAG 2.1 AA) for embedded chat interfaces.
Preferred: Experience with performance/load testing tools (k6, Locust, JMeter).
Preferred: Familiarity with Jira, Confluence, and Agile workflows.

Benefits

Comp & perks

Competitive salary and performance-based bonus.
Comprehensive health, dental, and vision benefits.
Generous PTO, holidays, and flexible working arrangements.
Annual learning and development budget.
Access to Pearson's full catalog of learning products and certifications.
Opportunity to shape quality practices across a globally recognized education technology organization.