AI Quality Assurance, Prompt Team

SitusAMC

. Define, execute, and continuously improve quality assurance standards for AI agent and prompt outputs, ensuring generated responses meet expectations for accuracy, consistency, completeness, and explainability.

Posted 5/22/2026full-timeRemote • 🇺🇸 United StatesMid-LevelSenior💰 $110,000 - $130,000 per yearWebsite

About the role

Key responsibilities & impact

Define, execute, and continuously improve quality assurance standards for AI agent and prompt outputs, ensuring generated responses meet expectations for accuracy, consistency, completeness, and explainability.
Oversee QA activities throughout the AI lifecycle, including pre‑deployment validation and post‑deployment output monitoring (regression testing).
Ensure AI outputs support informed decision-making by validating clarity, applicability, and alignment with intended business value.
Serve as an advocate for high-quality AI outputs, promoting shared understanding of output standards and expectations across the organization.
Lead and oversee QA Analysts responsible for reviewing and evaluating AI agent and prompt outputs.
Provide direction, coaching, and prioritization to ensure consistent application of output QA standards and timely completion of QA activities.
Forecast QA timelines and capacity in coordination with AI leadership to support delivery commitments and ensure deadlines are met.
Coordinate cross team to align QA activities with delivery timelines and ensure prompt and agent readiness.
Assist with the development of repeatable AI output evaluation approaches that can be applied across use cases to drive efficiency and consistency.
Document QA findings, trends, and recurring issues related to AI agent and prompt outputs.
Prepare reports, summaries, and presentations for management outlining QA results, risks, and improvement opportunities.
Identify opportunities to improve QA processes, tools, and standards based on observed output behavior and delivery feedback.

Requirements

What you’ll need

Bachelor’s degree in a relevant field (such as Business, Computer Science, Data Science, Engineering, or related discipline), or equivalent combination of education and experience.
Minimum of 6+ years of industry and/or relevant experience, typically with 1+ years in a Senior Associate level role or external equivalent.
8+ years of relevant professional experience, with demonstrated involvement in AI, analytics, content evaluation, automation, or quality assurance disciplines.
Prior QA management experience preferred, including oversight of analysts or reviewers responsible for evaluating outputs or deliverables.
Prior leadership experience preferred, with the ability to coordinate work, forecast timelines, and drive accountability across teams.
Demonstrated experience evaluating AI-generated outputs, including prompts, agent responses, or decision-support content.
Strong understanding of how prompt design, instructions, and context influence AI agent behavior and output quality.
Excellent written and verbal communication skills, with the ability to explain quality findings and recommendations to both technical and non-technical stakeholders.
Strong analytical and critical thinking skills, with attention to patterns, edge cases, and systemic quality issues.
Experience working in Agile or delivery-oriented environments, coordinating priorities across multiple stakeholders.
Real estate, mortgage, or financial services experience preferred, but not required.

Benefits

Comp & perks

PTO and paid holidays
Health insurance
Dental insurance
Vision insurance
Life insurance
Disability insurance
401K

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

quality assuranceAI evaluationregression testingdata analysisautomationprompt designdecision-support contentreportingprocess improvementanalytics

Soft Skills

leadershipcommunicationanalytical thinkingcritical thinkingcoachingcollaborationforecastingaccountabilityorganizationproblem-solving