AI Implementation Quality Analyst

Granicus

AI Quality & Evaluation Analyst responsible for ensuring AI-generated responses meet quality standards at Granicus. Partnering with clients and teams to align expectations and evaluate outputs effectively.

Posted 6/11/2026full-timeRemote • 🇺🇸 United StatesMid-LevelSenior💰 $80,000 - $105,700 per yearWebsite

About the role

Key responsibilities & impact

Partner directly with clients during implementation to understand use cases, success criteria, and risk tolerance
Translate client requirements into evaluation frameworks, prompt strategies, and test coverage
Act as the quality liaison between client, product, and engineering to ensure alignment pre- and post-launch
Review and score AI responses using standard rubrics
Validate against sources/ground truth; flag hallucinations, omissions, and other risks
Document findings and calibrate with peers to keep scoring consistent
Build and maintain prompt banks and golden sets (expected results)
Expand coverage for edge cases, high-risk scenarios, and real user language
Track regression/drift and feed dashboards and quality reports
Triage internal and client feedback; synthesize themes across deployments
Identify systemic risks and escalate high-impact findings with clear, client-relevant context
Partner with product/engineering to validate and verify fixes, ensuring alignment with client expectations

Requirements

What you’ll need

Strong analytical judgment and attention to detail
Excellent written communication skills, including the ability to explain reasoning clearly
Experience reviewing, auditing, or evaluating structured outputs such as content, decisions, or recommendations
Comfort applying detailed guidelines and rubrics consistently at scale
Familiarity with large language models and common failure modes such as hallucinations, overgeneralization, or unsafe responses
Ability to work in spreadsheets, evaluation tools, or annotation platforms
Experience with AI evaluation, data annotation, QA, trust and safety, or policy review
Exposure to human-in-the-loop workflows, or benchmarking processes
Domain expertise in a regulated or high-risk field such as government, education, healthcare, or legal services
Experience contributing to test suites, evaluation dashboards, or quality reporting
Comfort working cross-functionally with product and engineering teams
Experience in client-facing roles such as implementation, consulting, or solution delivery in a SaaS or regulated environment
Ability to translate between business/user needs and technical system behavior.

Benefits

Comp & perks

Flexible Time Off – Take the time you need to rest, recharge, and live your life.
Company-Wide Wellbeing Days – Paid days off to unplug and focus on your mental health.
Work From Home Reimbursement – Support a productive home office environment.
Multiple Health Plan Options – Including a 100% employer-paid plan.
Employer HSA Contributions – When enrolled in a High-Deductible Health Plan.
Fitness Reimbursement Program – Stay active, your way.
On-Demand Mental Health Support – Access to Headspace and other wellness tools.
Paid Parental Leave – For both birthing and non-birthing parents.
Traditional & Roth 401(k) – With a generous company match.
Life & AD&D Insurance – 100% employer-paid coverage for peace of mind.
Online Learning Platforms – Fuel your professional development.
Competitive Salary & Bonuses – Your contributions are valued and rewarded.

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

AI evaluationdata annotationquality assurancetest suitesevaluation dashboardsrubricsspreadsheetsevaluation toolsannotation platformshuman-in-the-loop workflows

Soft Skills

analytical judgmentattention to detailwritten communicationclient-facingcross-functional collaborationsynthesizing themesrisk identificationproblem-solvingdocumentationfeedback triage