Develop and maintain automated test frameworks for LLM, RAG, agentic, and classic ML systems
Implement golden/adversarial test cases (functional, safety, bias, prompt injection, refusal correctness, tool use)
Integrate evaluation tools (promptfoo, Ragas) into CI/CD pipelines and support batch quality assessments
Author and validate OpenAPI /JSON Schema contracts for model inputs/outputs; enforce schema checks in pipelines
Instrument observability metrics (latency, success rate, cost, drift) using OpenTelemetry and export to dashboards
Support data QA, lineage tracking, and drift monitoring (Evidently, OpenLineage)
Collaborate with AI developers, SDETs, and SRE to troubleshoot, optimize, and document automation processes
Contribute to red-team playbooks and security testbooks mapped to OWASP/ATLAS risks

Requirements

Experience in test automation, software engineering, or quality assurance (AI/ML experience is a plus, but not required)
Proficiency in Python, JavaScript/Node.js, or similar programming languages
Familiarity with CI/CD, version control, and cloud-based deployment environments
Experience with test automation tools (promptfoo, Ragas, Postman)
Understanding of AI model evaluation metrics (faithfulness, answer relevancy, hallucination, refusal correctness)
Basic knowledge of fairness, safety, and security testing for AI systems
Experience with AI infrastructure, orchestration, data integrity, deployments, and model registry practices
Interest in Responsible AI, bias mitigation, and regulatory compliance
Bachelor’s degree in Computer Science, Engineering, or related field, or equivalent practical experience

Benefits

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

test automationsoftware engineeringquality assurancePythonJavaScriptNode.jsAI model evaluation metricsfairness testingsafety testingsecurity testing

collaborationtroubleshootingoptimizationdocumentation