Senior QA Automation Engineer, AI & LLM Testing

Nagarro

Senior QA Automation Engineer designing and maintaining test automation for AI applications at Nagarro. Collaborating on projects and improving quality practices in Agile environments.

Posted 6/25/2026full-timeRemote • 🇶🇦 QatarSeniorWebsite

Tech Stack

Tools & technologies

AndroidiOSPython

About the role

Key responsibilities & impact

Design, develop, and maintain automated test suites for web applications, APIs, and AI-powered features
Evaluate LLM outputs for response accuracy, relevance, consistency, and hallucination risks
Create prompt tests and quality checks using Promptfoo or similar LLM evaluation frameworks
Build end-to-end UI automation using Playwright, following page object models and maintainable test architecture principles
Develop reusable Python scripts for API testing, automation utilities, and LLM quality evaluation
Create and maintain Postman collections for REST API testing, including environments, assertions, and CI/CD integration
Use GitHub Copilot or similar AI coding assistants to improve test coverage, speed up automation development, and identify quality gaps
Collaborate closely with product managers, developers, and stakeholders to define test strategies, communicate risks, and improve quality practices
Actively contribute to Agile ceremonies, including refinement sessions, sprint planning, retrospectives, and daily collaboration

Requirements

What you’ll need

Strong experience in QA Automation and software testing - 5+ years
Hands-on experience testing AI or LLM-based features, including prompt testing, output validation, response accuracy, and hallucination mitigation
Experience with Promptfoo or similar LLM evaluation tools/frameworks
Strong proficiency in Playwright for end-to-end UI automation
Solid knowledge of page object patterns, reusable test components, and scalable test architecture
Strong Python scripting skills for automation, API helpers, and custom quality checks
Experience with Postman and REST API testing, including collections, assertions, environments, and CI/CD integration
Practical experience using GitHub Copilot or similar AI coding assistants
Experience working in Agile environments such as Scrum or Kanban
Strong communication skills, with the ability to explain test strategies and quality risks to technical and non-technical stakeholders

Benefits

Comp & perks

Flexible work arrangements
Professional development opportunities

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

QA Automationsoftware testingAI testingLLM evaluationPrompt testingPlaywrightPython scriptingPostmanREST API testingCI/CD integration

Soft Skills

communicationcollaborationrisk managementAgile methodologiesproblem-solving