Netflix

Human Evaluation Program Manager

Netflix

full-time

Posted on:

Location Type: Remote

Location: United States

Visit company website

Explore more

✨ AI Apply
Apply

Salary

💰 $230,000 - $340,000 per year

About the role

  • Lead end-to-end execution of human evaluation and data operations initiatives—from intake and scoping to delivery
  • Develop and operationalize frameworks for evaluating GenAI and ML outputs
  • Collaborate across research, product, UX, and engineering to embed evaluation into model development cycles
  • Build and maintain project timelines, proactively manage blockers, and ensure timely execution
  • Develop clear, scalable guidelines and scoring rubrics to ensure consistent rater judgment
  • Oversee rater onboarding, calibration, and QA workflows
  • Define and monitor success metrics such as speed to IRR, throughput, and task effectiveness
  • Pilot and refine evaluation tasks to improve clarity, inter-rater reliability, and feedback quality
  • Build foundational documentation and drive adoption of best practices across teams
  • Track evaluation health and proactively communicate progress to stakeholders clearly and proactively
  • Anticipate and proactively resolve bottlenecks and blockers
  • Act as the connective tissue across multiple partners to ensure alignment and effective execution of evaluations at scale

Requirements

  • 4+ years of experience working in human evaluations, data collection, labeling, or annotation operations in GenAI/ML environments
  • Track record of implementing process improvements or quality control systems for data collection needs
  • Prior experience managing human annotation vendors, raters, or data labeling teams
  • Strong understanding of evaluation design, including guidelines, rubrics, and scoring protocols
  • Proven ability in end-to-end management of complex, cross-functional programs, demonstrating strong Program Management skills and clear accountability for successful delivery
  • Experience with human labeling platforms
  • Excellent written and verbal communication skills
  • Ability to synthesize feedback into clear recommendations and process improvements
  • Familiarity with responsible AI principles and how to embed them into evaluation design
  • Strong organizational skills and executional focus; ability to track details while seeing the bigger picture.
Benefits
  • 📊 Check your resume score for this job Improve your chances of getting an interview by checking your resume score before you apply. Check Resume Score
Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools
human evaluationdata operationsevaluation designprocess improvementsquality control systemsscoring protocolscross-functional program managementdata labelingevaluation metricsinter-rater reliability
Soft Skills
communication skillsorganizational skillsexecutional focusproblem-solvingcollaborationsynthesis of feedbackproactive managementaccountabilityadaptabilitystakeholder management