QA Engineer, AI

The Mortgage Office

QA Engineer responsible for validating AI systems and software testing at Applied Business Software, Inc. Working on end-to-end product flows and evaluating AI-driven workflows.

Posted 4/12/2026full-timeSan Mateo • California • 🇺🇸 United StatesMid-LevelSenior💰 $120,000 - $140,000 per yearWebsite

Tech Stack

Tools & technologies

Python

About the role

Key responsibilities & impact

Test end-to-end product flows across frontend, backend, and APIs of an event based system
Design structured test cases for prompt-based features
Build and maintain test datasets
Define evaluation criteria for AI outputs
Run regression testing when prompts or models change
Automate repeatable tests
Track quality metrics over time (output quality, failure rate, latency issues)

Requirements

What you’ll need

3+ years in QA / software testing, preferably for LLM based product
Strong Python skills (Automation, LLM, Evaluations)
Experience testing APIs and full-stack systems
Ability to think in edge cases and break systems intentionally
Experience working with LLM-based products
Experience testing prompt engineering products.

Benefits

Comp & perks

Comprehensive health, dental, and vision coverage.
401(k) with company funded match.
Competitive compensation and room for professional growth.
Performance-based bonuses.
Generous PTO.
Modern workspace with on-site amenities.
Awesome company Christmas party.

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

PythonAutomationAPI testingSoftware testingRegression testingTest case designTest dataset managementQuality metrics trackingLLM evaluationPrompt engineering

Soft Skills

Critical thinkingProblem-solvingAttention to detailAnalytical skills