QA Analyst – Quality & AI Agents

Mutant

QA Engineer ensuring quality, reliability, and monitoring for AI conversational agents. Collaborating closely with technical teams to improve agent responses and behavior.

Posted 7/8/2026full-timeSão Paulo • 🇧🇷 BrazilMid-LevelSeniorWebsite

Tech Stack

Tools & technologies

Python

About the role

Key responsibilities & impact

Ensure the quality, reliability, and consistency of responses from LLM-based agents
Design test strategies, validation automation, and monitoring for agent behavior
Create test strategies for conversational agents based on LLMs
Develop test scenarios to validate agent behavior
Automate tests for prompts and model responses
Identify hallucinations, inconsistencies, and inappropriate responses
Create test datasets and regression test suites
Validate adherence to guardrails and response policies
Participate in quality reviews prior to deployment of new agent versions
Work with technical teams to correct and improve agent behavior
Monitor response quality in production

Requirements

What you’ll need

Experience in Quality Assurance or Test Automation
Experience creating test scenarios and test plans
Ability to structure systematic tests for complex systems
Experience with APIs and automated testing
Hands-on experience with AI tools such as ChatGPT, Claude, Gemini
Analytical and investigative mindset
Strong attention to detail
**Strong advantages**
Experience with Generative AI or LLMs
Experience testing chatbots or voicebots
Knowledge of NLP or text analysis
Experience with test automation in Python
Experience in Customer Experience or Contact Centers
Experience testing AI-based systems

Benefits

Comp & perks

Health and dental insurance
Life insurance
Transportation allowance
Meal and food allowance
Wellhub (Gympass)
Childcare assistance
Assistance for children with disabilities
Performance-based bonus
Birthday day off
Alelo Multibenefits
Allya – partnerships and discounts via the app
Pet health plan

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

Test ScenariosTest PlansAutomated TestingNLPText AnalysisChatbot TestingVoicebot TestingAnalytical MindsetAttention To DetailRegression Test Suites

Soft Skills

Investigative Mindset