AI Evaluation Program Manager

Twelve Labs

AI Evaluation Program Manager at Twelve Labs focused on designing and building model evaluation frameworks. Leading data operations projects to enhance video understanding and multimodal AI capabilities.

Posted 5/7/2026full-timeSan Francisco • California • 🇺🇸 United StatesMid-LevelSenior💰 $150,000 - $160,000 per yearWebsite

Tech Stack

Tools & technologies

Python

About the role

Key responsibilities & impact

Design and build robust model evaluation frameworks, automating repetitive processes and maintaining a balanced approach to efficiency and depth in obtaining evaluation metrics and feedback.
Manage resource allocation and timelines, adjusting direction flexibly based on real-time information across all data streams in your product vertical.
Enhance dataset and process quality through seamless collaboration with vendors and outsourcing partners.
Establish labeling guidelines, monitor data quality, and improve tools and infrastructure to build a sustainable data operations framework.
Partner with Engineering and AI Model teams to align on top priority data needs, design tools such as analytical reports and dashboards, and clearly communicate project progress.

Requirements

What you’ll need

5+ years of experience working in an AI focused data operations organization.
A proven track record designing and executing large scale data or evaluation projects, including gathering, labeling, and post-processing data.
The ability to analyze messy and complex data, identify overarching patterns, and distill your findings into crisp annotation guidelines or model quality reports.
Proficiency with Python, LLMs, or other popular industry tools for automation.
Excellent communication and project management skills, and the ability to support several projects simultaneously.
A foundational understanding of and interest in LLMs/VLMs and multimodal AI.
Conviction that data is the key ingredient for the performance and assessment of AI models.

Benefits

Comp & perks

Full health, dental, and vision benefits.
Flexible PTO and parental leave policy.
Office closed the week of Christmas and New Years.

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

Pythondata evaluationdata labelingdata post-processingdata analysisautomationLLMsVLMsmultimodal AImodel quality reporting

Soft Skills

communicationproject managementcollaborationflexibilityanalytical thinkingresource allocationtime managementproblem-solvingattention to detailadaptability