Senior Product Operations Manager, Evaluation Quality

Harvey

Senior Product Operations Manager setting quality standards for Harvey’s human evaluations. Certifying reliable legal AI evaluation data for Product, Engineering, and AI Research.

Posted 8/4/2026full-timeSan Francisco • California • 🇺🇸 United StatesSenior💰 $155,400 - $233,200 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in quality program management, evaluation design, and inter-rater reliability, with a strong focus on developing and maintaining evaluation guidelines and benchmarks. Proven ability to analyze evaluation data and implement feedback loops to enhance reviewer quality across diverse jurisdictions.

Highest-signal resume keywords

Quality Program ManagementEvaluation DesignInter-Rater ReliabilityOnboarding and TrainingCross-Functional Coordination

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Evaluation GuidelinesBenchmark DevelopmentQA ProceduresRubric DesignData AnalysisCalibration TrainingFeedback LoopsError TaxonomiesOperational DashboardsHuman Evaluation

Soft Skills

Clear CommunicationStakeholder CredibilityHigh OwnershipBias to Action

Industry Keywords

Legal TechRegulated IndustryHuman-in-the-LoopDistributed Expert RatersMultiple MarketsNon-English GeographiesHyper-Growth Startup

About the role

Key responsibilities & impact

Own the quality bar for Harvey’s human evaluations and make outputs decision-grade for Product, Engineering, and AI Research
Author and maintain evaluation guidelines, instructions, benchmarks, and gold references for contract attorneys
Arbitrate ambiguous or disputed judgments
Standardize rubric and evaluation design into repeatable templates and a documented methodology
Own contract-attorney onboarding, calibration training, inter-rater reliability, and quality feedback loops
Define and maintain failure-mode and error taxonomies
Run QA on vendor and contract-attorney deliverables before launch decisions
Ensure quality across jurisdictions and non-English geographies
Establish a standard method for analyzing evaluation results
Build operational dashboards tracking rater capability and evaluation-program health
Support Applied Legal Research in certifying evaluations before scaling them to contract attorneys
Partner with Applied Legal Research and Analytics to compare human evaluations with online signals and identify expanded insights

Requirements

What you’ll need

6+ years in product operations, research operations, evaluation/QA operations, or quality program management
Track record owning quality inputs such as guidelines, instructions, benchmarks, and QA procedures for complex expert-driven or human-in-the-loop work
Experience onboarding, training, and calibrating distributed expert raters, annotators, or reviewers
Experience running feedback loops to improve reviewer quality over time
Grounding in calibration, inter-rater reliability, sampling, and rubric design
Ability to interpret evaluation data, natively or with AI tool support
Experience scaling and streamlining quality processes under shipping pressure
Preference for documentation and reproducibility over one-off analysis
Ability to work deeply with domain experts and translate nuanced judgment into repeatable documented standards
Strong cross-functional coordination across Product, Engineering, Research, Applied Legal Research, and vendors
Clear communication and ability to build stakeholder credibility and trust
High ownership and bias to action
Legal tech or regulated-industry experience is a bonus
Experience across multiple markets, languages, or jurisdictions is a bonus
Experience building calibration, inter-rater reliability, or capability-tracking systems is a bonus
Experience transitioning evaluation work in-house or improving evaluation ROI is a bonus
Familiarity with LLM-as-judge or automated evaluation is a bonus
Hyper-growth startup or world-class product/platform operations experience is a bonus
Ability to work from the San Francisco office 3 days per week
Must be legally authorized to work in the country where the role is located

Benefits

Comp & perks

Equity plan
Benefits program
Comprehensive health coverage
Dental coverage
Vision coverage
Retirement benefits with 401(k) match up to 4%
Flexible PTO
Opportunity for personal, professional, and financial growth
Reasonable accommodations for applicants with disabilities