Content Merchandising Program Lead
Tubi
Executive Director, Natural Language Measurement
The College Board
full-time
Posted on:
Origin: • 🇺🇸 United States
Visit company websiteSalary
💰 $148,000 - $225,000 per year
Job Level
Lead
About the role
- Shape and implement the multi-year strategy and roadmap for automated scoring and intelligent learning approaches across SAT, AP, CLEP, Accuplacer and future products.
- Advance College Board’s innovation agenda by expanding from automated scoring to broader AI-enabled solutions that support personalized learning and actionable feedback.
- Establish model-readiness criteria and guide release decisions for senior leadership to ensure high-quality, scalable deployment of natural language-driven solutions.
- Partner closely with Technology to shape development of guided scoring products and other intelligent learning innovations.
- Lead scientific design, implementation, and evaluation of automated scoring and AI/NL applications, ensuring alignment with program and organizational goals.
- Apply psychometric and measurement principles to ensure all AI/NL-driven applications uphold validity, fairness, reliability, and equity.
- Guide implementation of initial automated scoring programs beginning in 2026 and establish operational processes including monitoring, human-in-the-loop procedures, and evaluation criteria.
- Build organizational readiness for adoption of new technologies and establish criteria for model evaluation, quality assurance and responsible release decisions.
- Build and lead a team of psychometricians, language model experts, and related specialists; set vision and priorities, track progress, provide coaching, and cultivate an inclusive high-achieving culture.
- Serve as College Board’s lead subject-matter expert on AI powered natural language models; represent the organization externally, communicate responsible use guidance, and collaborate across divisions.
Requirements
- At least 10 years of experience in psychometrics, assessment, or measurement-related fields, with a strong record of applying measurement science in large-scale educational contexts and leading interdisciplinary innovation across psychometrics, AI, and product development.
- Deep expertise in measurement science and familiarity with AI/NL approaches, with the ability to apply principles of validity, reliability, fairness, and equity in large-scale assessments.
- Proven ability to move AI powered natural language systems from research into operational use, including shaping automated scoring strategies and collaborating with technology and operations partners to ensure quality, fairness, efficiency and integrity.
- Demonstrated success setting strategy and guiding innovation through cross-functional collaboration, leading teams across psychometricians, data scientists, engineers, and program staff to execute complex initiatives at scale.
- At least 5 years of experience leading and managing direct teams with diverse expertise in measurement, technology, and product, with a strong track record of hiring, developing, and retaining high-performing talent.
- Doctorate in psychometrics, educational measurement, statistics, psychology, or a related field.
- Exceptional communication and organizational influence skills.
- Deep commitment to responsible use of artificial intelligence and to advancing equity in educational measurement.
- A strategic and inclusive leadership style and proven ability to drive performance and growth.
- The ability to travel 10-12 times a year to College Board offices or on behalf of College Board business.
- Authorization to work in the United States for any employer.
