QA Engineer

Titan AI

QA Engineer developing evaluation frameworks for AI banking software. Ensuring quality through test coverage, compliance, and CI/CD processes.

Posted 5/27/2026full-timeRemote • 🇺🇸 United StatesSeniorLeadWebsite

Tech Stack

Tools & technologies

AzurePythonSelenium

About the role

Key responsibilities & impact

Titan builds AI software for banks: purpose-built small language models, a banking ontology, and AI bankers that financial institutions can trust.
You are coming in to do the work: write the test cases, build the evaluation framework, set up CI/CD gates, and triage bugs alongside engineering.
You personally design and execute the evaluation framework for LLM and agentic AI outputs across Foundry, Agent Builder, and client-deployed instances.
You write the assertions, define the behavioral contracts, and own regression baselines for model behavior.
You write and maintain the automated test suite: end-to-end, integration, and regression coverage for backend APIs, document ingestion pipelines, AI inference workflows, and frontend surfaces.
You produce the test artifacts, audit logs, and process documentation that meet SOC 2 Type II standards.

Requirements

What you’ll need

Seven or more years in software QA engineering, with at least two years personally testing AI or ML systems.
You have written test cases against LLM outputs, built evaluation pipelines from scratch, and know the difference between a flaky test and a genuinely non-deterministic system.
You are fluent in Python and have built automated suites using pytest, Playwright, or Selenium.
You have hands-on experience with RAGAS, DeepEval, LangSmith, or comparable evaluation tooling—not just familiarity with the names.
You can trace a failure from the application layer to infrastructure and know enough about Azure, async systems, and REST APIs to do it without waiting on an engineer to walk you through it.
You have integrated QA gates into CI/CD pipelines and owned the process end to end.
Experience in fintech, banking, or another regulated environment is a strong advantage.
Familiarity with document processing pipelines, multi-agent architectures, RAG validation, or observability tooling such as Arize or Langfuse puts you ahead.

Benefits

Comp & perks

Competitive base and meaningful equity.
Remote (US). Occasional travel to client sites and team offsites.

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

software QA engineeringtest case writingevaluation framework designautomated test suite maintenanceend-to-end testingintegration testingregression testingPythonpytestSelenium

Soft Skills

problem-solvingattention to detailcommunicationcollaborationcritical thinking

Certifications

SOC 2 Type II compliance