Head of Data Labeling

White Circle

Head of Data Labeling overseeing data annotation and AI evaluation projects for an AI Safety company. Leading a team in defining standards and improving workflows and quality assurance processes.

Posted 7/8/2026full-timeParis • 🇫🇷 FranceLead💰 $60,000 - $100,000 per yearWebsite

Tech Stack

Tools & technologies

PythonSQL

About the role

Key responsibilities & impact

Build from scratch and lead the Data Labeling team (hiring, coaching, and performance management)
Define annotation guidelines, quality standards, and evaluation frameworks
Develop quality assurance processes, calibration sessions, and auditing systems
Partner with AI researchers and engineers to translate research objectives into labeling workflows
Prioritise labeling projects based on business and research needs
Monitor operational metrics including quality, consistency, throughput, and cost
Improve annotation tooling, automation, and workflow efficiency
Lead complex AI evaluation projects, including safety, preference ranking, RLHF, policy evaluation, and benchmark creation
Analyse disagreement patterns and edge cases to improve guidelines and model performance
Manage vendor relationships and ensure consistent quality across distributed teams
Build reporting dashboards and communicate operational insights to leadership
Foster a culture of continuous improvement, accountability, and operational excellence

Requirements

What you’ll need

Has experience leading data annotation or AI evaluation teams
Has strong operational and people management skills
Understands AI model evaluation, LLM behavior, and modern annotation workflows
Can design scalable processes without sacrificing quality
Communicates clearly across technical and non-technical teams
Thrives in fast-moving startup environments
Have managed annotation programs for LLMs, generative AI, or machine learning
Have experience with RLHF, preference data collection, safety evaluations, or benchmark creation
Have worked in Trust & Safety, AI Safety, Content Moderation, or ML Ops
Have managed distributed or global annotation teams
Have experience with vendor management and outsourcing operations
Familiarity with prompt engineering and AI safety policies
SQL, Python, or data analysis experience
Experience building internal annotation platforms or workflow automation
Background in linguistics, cognitive science, machine learning, or data operations

Benefits

Comp & perks

Competitive salary + equity
Work from Paris (hybrid) with a relocation package available, or work from London (note: we are currently unable to provide relocation support and medical insurance for London-based roles)
Paid time off in line with your local regulations
All the hardware, tools, and services you need
Covered subscriptions for AI agents and IDEs
Team off-sites twice a year: we’ve recently been to the Alps and Saint-Tropez

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

Data AnnotationAI EvaluationQuality Assurance ProcessesWorkflow AutomationPerformance ManagementData AnalysisScalable Process DesignBenchmark CreationRLHF ExperienceAnnotation Tooling Improvement

Soft Skills

Clear CommunicationCoachingContinuous ImprovementAccountabilityTeam Management