Senior QA Automation Engineer, AI & LLM Testing

Nagarro

Senior QA Automation Engineer at Nagarro designing automated test suites and evaluating AI-powered features. Collaborating with teams in Agile environment for high-quality solutions.

Posted 7/15/2026full-timeRemote • 🇶🇦 QatarSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates strong expertise in QA Automation, particularly in testing AI and LLM-based features, with proficiency in Playwright for UI automation and Python scripting for API testing. Capable of collaborating effectively in Agile environments while communicating test strategies and quality risks to diverse stakeholders.

Highest-signal resume keywords

QA AutomationPlaywright UI AutomationPython ScriptingPostman API TestingAgile Methodologies

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Automated Test SuitesLLM EvaluationPrompt TestingEnd-to-End TestingReusable Test ComponentsCI/CD IntegrationOutput ValidationResponse AccuracyHallucination MitigationMobile Test Automation

Soft Skills

Strong Communication

Tools & Technologies

PromptfooGitHub CopilotPostmanAgile Ceremonies

Industry Keywords

AI TestingLLM FeaturesPage Object ModelTest ArchitectureScrumKanban

Tech Stack

Tools & technologies

AndroidiOSPython

About the role

Key responsibilities & impact

Design, develop, and maintain automated test suites for web applications, APIs, and AI-powered features.
Evaluate LLM outputs for response accuracy, relevance, consistency, and hallucination risks.
Create prompt tests and quality checks using Promptfoo or similar LLM evaluation frameworks.
Build end-to-end UI automation using Playwright, following page object models and maintainable test architecture principles.
Develop reusable Python scripts for API testing, automation utilities, and LLM quality evaluation.
Create and maintain Postman collections for REST API testing, including environments, assertions, and CI/CD integration.
Use GitHub Copilot or similar AI coding assistants to improve test coverage, speed up automation development, and identify quality gaps.
Collaborate closely with product managers, developers, and stakeholders to define test strategies, communicate risks, and improve quality practices.
Actively contribute to Agile ceremonies, including refinement sessions, sprint planning, retrospectives, and daily collaboration.

Requirements

What you’ll need

Strong experience in QA Automation and software testing.
Hands-on experience testing AI or LLM-based features, including prompt testing, output validation, response accuracy, and hallucination mitigation.
Experience with Promptfoo or similar LLM evaluation tools/frameworks.
Strong proficiency in Playwright for end-to-end UI automation.
Solid knowledge of page object patterns, reusable test components, and scalable test architecture.
Strong Python scripting skills for automation, API helpers, and custom quality checks.
Experience with Postman and REST API testing, including collections, assertions, environments, and CI/CD integration.
Practical experience using GitHub Copilot or similar AI coding assistants.
Experience working in Agile environments such as Scrum or Kanban.
Strong communication skills, with the ability to explain test strategies and quality risks to technical and non-technical stakeholders.
Nice to have: Experience with mobile test automation using Appium, XCUITest, or similar tools.
Understanding of iOS and Android platform-specific testing considerations.
Experience integrating automated tests into CI/CD pipelines.
Familiarity with MCP or agent-assisted testing approaches in Playwright.

Benefits

Comp & perks

Employees can work remotely