Quality Assurance Automation Engineer – AI

OnePlan

QA Automation Engineer specializing in AI-driven features testing at OnePlan. Collaborating with cross-functional teams to ensure quality, reliability, and performance of AI capabilities.

Posted 7/2/2026full-timeRemote • 🇨🇦 CanadaJuniorMid-LevelWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in QA Automation Engineering with a focus on building and maintaining automated test frameworks for AI features. Proficient in testing AI/ML-integrated services, ensuring quality through robust test strategies and collaboration with cross-functional teams.

Highest-signal resume keywords

QA Automation EngineeringAutomation Tools (Playwright, Claude)API Testing (Postman, REST-assured)AI/ML Feature TestingAnalytical and Critical-Thinking Skills

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Automation Framework DevelopmentTest Strategy DevelopmentFunctional TestingRegression TestingPerformance TestingScalability TestingDefect TrackingTest Case DesignCI/CD IntegrationAI Quality Practices

Soft Skills

Verbal CommunicationWritten CommunicationCollaborationCritical Thinking

Tools & Technologies

PlaywrightClaudePostmanREST-assured

Industry Keywords

AI FeaturesLLM OutputsRecommendation EnginesIntelligent WorkflowsNon-Deterministic OutputsProbabilistic RecommendationsGenerative ContentQuality StandardsAcceptance CriteriaAI Safety Checklists

About the role

Key responsibilities & impact

Design, build, and maintain robust automation frameworks specifically tailored for testing AI features, including LLM-powered outputs, recommendation engines, and intelligent workflows.
Develop and execute test strategies for validating AI model outputs, including accuracy, relevance, consistency, and bias evaluation across diverse input scenarios.
Perform functional, regression, performance, and scalability testing on AI-driven features, including API-level testing of AI service integrations and end-to-end automated test suites.
Collaborate with AI engineers, data scientists, product managers, and UX designers to define acceptance criteria and quality standards for AI features throughout the development lifecycle.
Identify, document, and track defects in AI behavior, including edge cases, hallucinations, and unexpected model responses through resolution using appropriate tools and workflows.
Champion quality and responsible AI practices throughout the development lifecycle, contributing to prompt engineering reviews, model evaluation rubrics, and AI safety checklists.
Build and maintain CI/CD-integrated automation pipelines that continuously validate AI feature quality, using tools such as Playwright, Claude, or equivalent frameworks.
Participate in team stand-ups, sprint planning, retrospectives, and continuous improvement activities, bringing an AI-quality-first mindset to every discussion and helping establish best practices for testing intelligent systems.

Requirements

What you’ll need

2-4 years of experience in QA automation engineering, with demonstrated hands-on work building and maintaining automated test frameworks.
Proficiency with automation tools and frameworks (e.g., Playwright, Claude, or similar) and API testing tools (e.g., Postman, REST-assured).
Demonstrated experience testing AI/ML-integrated features or services, with a solid understanding of the unique challenges of validating non-deterministic outputs, including LLM responses, probabilistic recommendations, and generative content.
Bachelor’s degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience.
Strong analytical and critical-thinking skills, with the ability to design test cases for complex, probabilistic AI behaviors.
Excellent verbal and written communication skills, with the ability to clearly articulate AI quality issues and their business impact to both technical and non-technical stakeholders.

Benefits

Comp & perks

We’re a remote-first company with team members across the USA, Canada, UK, and India!
OnePlan has been recognized as the Global Microsoft Partner of the Year in Project Portfolio Management in 2019, 2020, 2021, 2022 and 2023.
We’ve been named a "Strong Performer" in the latest Forrester Strategic Portfolio Management WAVE report.
We offer comprehensive health, dental, and vision benefits, with additional insurance options.
Employer RRSP and 401K matching programs.
A fun, collaborative, and diverse environment with regular health and team challenges to keep things light and enjoyable!