Head of Data Science – Product Experimentation, Machine Learning
ItsaCheckmate
full-time
Posted on:
Location Type: Remote
Location: United States
Visit company websiteExplore more
Job Level
About the role
- Own end-to-end product experimentation: hypothesis generation, metric definition, experimental design (A/B, multivariate, sequential testing), analysis, and executive-level interpretation.
- Design and maintain ML-powered evaluation frameworks for product changes, automation quality, and system reliability (e.g., order accuracy, routing, error rates, conversion).
- Build and deploy predictive models, classifiers, and ranking systems that power experimentation, personalization, and product optimization.
- Partner with product and engineering to test new features, workflows, and ML models through controlled experiments and incremental rollouts.
- Lead offline and online model evaluation, comparing baselines, candidate models, and product variants using rigorous statistical methods.
- Use causal inference and quasi-experimental methods when randomized experiments are not feasible.
- Develop experiment pipelines and instrumentation: logging, dashboards, monitoring, and automated analysis to ensure measurement integrity.
- Perform failure-mode and error analysis to guide product iteration and model improvement.
- Translate experiment outcomes into clear product decisions, influencing roadmap prioritization and system design.
- Drive experimentation at scale in a fast-moving environment, balancing speed, rigor, and business impact.
- Lead and mentor data scientists and analysts, setting standards for experimentation, modeling, and evaluation across the organization.
Requirements
- 8–12+ years of experience in data science, machine learning, or applied experimentation roles.
- Demonstrated expertise in product experimentation and A/B testing, including design, execution, and statistical evaluation.
- Strong background in machine learning, statistical modeling, and causal inference applied to real-world products.
- Experience building and evaluating predictive models, classifiers, or ranking systems in production environments.
- Proven ability to operate in both startup-style experimentation and scaled product ecosystems.
- Experience leading teams, setting technical direction, and delivering cross-functional impact.
- Excellent coding skills in Python (or similar), strong SQL, and experience building data pipelines or ML systems.
- Ability to connect technical findings to product and business outcomes.
- Strong communication skills with technical and non-technical stakeholders.
Benefits
- Health Care Plan (Medical, Dental & Vision)
- Retirement Plan (401k)
- Life Insurance (Basic, Voluntary & AD&D)
- Flexible Paid Time Off
- Family Leave (Maternity, Paternity)
- Short Term & Long Term Disability
- Training & Development
- Work From Home
- Stock Option Plan
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills & Tools
product experimentationA/B testingmultivariate testingsequential testingmachine learningstatistical modelingcausal inferencepredictive modelsdata pipelinesSQL
Soft Skills
leadershipmentoringcommunicationcross-functional collaborationanalytical thinkingproblem-solvinginfluencingdecision-makingadaptabilityteamwork