Develop comprehensive test strategies: Create and maintain test plans covering functional, performance, and regression testing for AI agents and automation workflows.
Build and maintain automated test frameworks: Develop and maintain automated test scripts/frameworks to ensure bot responses and workflow outputs remain accurate, consistent, and reliable over time.
Build and maintain test assets: Create/curate golden conversation sets, synthetic test data, evaluation rubrics, and regression suites to measure quality over time.
Validate AI agent safety and reliability: Test tool-calling end-to-end (tool selection, parameter mapping, sequencing, retries/timeouts, error handling, and recovery), including negative/edge cases; confirm outputs align to business rules and document risk mitigations.
Test integration/workflow integrity: Validate API contracts, data transformations, and downstream impacts (idempotency, duplicate prevention, partial failures, and latency) across environments.
Manage defects through resolution: Identify, log, prioritize, and track defects; partner with developers to reproduce issues and validate fixes.
Provide release certification: Deliver final quality sign-off prior to production release, ensuring defined acceptance criteria and quality standards are met.
Align on acceptance criteria: Partner with Business Analysts to ensure user stories include clear, testable acceptance criteria and that test coverage maps to requirements.
Document testing and metrics: Maintain test cases, data, results, and quality metrics; communicate outcomes and risks.
Participate in Agile ceremonies: Engage in planning, grooming, standups, and retrospectives to embed quality throughout delivery.
Drive continuous improvement: Identify opportunities to improve automation coverage, testing effectiveness, and quality practices.

Requirements

3+ years in a QA, test automation, or software testing role with demonstrated experience developing and executing test strategies.
Test automation expertise: Strong hands-on experience building automated test frameworks using tools such as Selenium, Playwright, Cypress, Jest, Pytest, or similar frameworks.
Scripting and programming skills: Proficiency in at least one programming or scripting language (Python, JavaScript, Java, C#) for test automation development.
Agile methodology: Direct experience working in an Agile/Scrum environment with hands-on involvement in sprint cycles and continuous integration.
Defect management tools: Experience using bug tracking and test management tools such as Jira, Azure DevOps, TestRail, or similar platforms.
Bachelors level degree is preferred, but not required.

Benefits

Health insurance
401(k) matching
Flexible work hours
Paid time off
Remote work options

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard skills

test strategiesautomated test frameworkstest scriptsfunctional testingperformance testingregression testingAPI validationdefect managementscriptingprogramming

Soft skills

communicationcollaborationproblem-solvingattention to detailcontinuous improvement