Senior QA Automation Engineer, AI Systems

Software Mind

Senior QA Automation Engineer focusing on AI systems within a multicultural team at Software Mind. Ensuring AI output quality and test framework design for complex RAG implementations.

Posted 6/17/2026full-timeRemote • 🇶🇦 QatarSeniorWebsite

Tech Stack

Tools & technologies

Python

About the role

Key responsibilities & impact

Design and implement the validation harness for RAG output quality: retrieval accuracy, citation correctness, and grounding
Build automated test suites for the AI Extraction Gateway across Simple RAG and Complex RAG implementations
Develop and execute accuracy rubric test cases in collaboration with the BA and Designated Subject Matter Expert
Automate regression testing for confidence score calibration and source weighting behaviour
Test RBAC enforcement and role-specific filtered view access controls
Validate audit logging completeness and document lifecycle traceability
Build and maintain the incremental ingestion pipeline test suite
Contribute to go/no-go decision packs: produce accuracy reports and test evidence documentation

Requirements

What you’ll need

6+ years in QA automation engineering; senior seniority required
Strong test automation engineering skills
Python preferred (pytest or equivalent framework)
Experience with API testing and contract testing
Comfortable designing test frameworks for non-deterministic or probabilistic systems
Experience in agile/scrum environments with Jira-based test management

Benefits

Comp & perks

Flexible work arrangements

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

QA automation engineeringtest automation engineeringPythonpytestAPI testingcontract testingtest frameworksregression testingaccuracy rubric test casesincremental ingestion pipeline

Soft Skills

collaborationdecision makingdocumentation