QA Engineer

• Design and execute test strategies for LLM-powered conversational flows, including response accuracy, relevance, assumption detection, and hallucination prevention.
• Build and maintain evaluation datasets to measure assistant performance against target metrics (95% accuracy, 75% CSAT).
• Validate context injection — ensure the assistant correctly receives and uses runtime context (role, product, LMS type, browser, workflow state, structured error/diagnostic codes) to tailor responses.
• Test the context-sharing contract between front-end applications and the AI Support Assistant — verifying session storage writes, schema compliance, and read interfaces.
• Validate cross-platform behavior across major LMS integrations (e.g., Canvas, Blackboard, Moodle, D2L) and multiple product lines across the courseware portfolio.
• Test entitlement and enrollment and third-party content-provider provisioning flows.
• Build automated regression suites for assistant response quality, context propagation, and UI behavior (chat widget placement, discoverability, mobile responsiveness).
• Develop performance and load testing strategies for projected scale (millions of interactions annually, growing year over year).
• Establish monitoring and alerting for production assistant accuracy, escalation rates, and context-pass-through failures.
• Partner with ML/AI engineers to define acceptance criteria for model updates and prompt changes.
• Work closely with product management to translate user research findings (e.g., accuracy as the #1 trust driver, step-based answers over article links) into testable requirements.
• Coordinate with backend platform and Customer Success teams on dependency validation.

QA Engineer jobs in Sri Lanka

See all jobs on JobTailor

Senior Quality Assurance Engineer