Staff AI Research Scientist – Evaluation
Handshake
PythonPyTorch
Staff AI Research Scientist, Data Quality
Handshake
full-time
Posted on:
Location Type: Office
Location: San Francisco • California • 🇺🇸 United States
Visit company websiteSalary
💰 $350,000 - $420,000 per year
Job Level
Lead
Tech Stack
PythonPyTorch
About the role
- Lead high-impact research on data quality frameworks for post-training LLMs — including techniques for preference consistency, label reliability, annotator calibration, and dataset auditing.
- Design and implement systems for identifying noisy, low-value, or adversarial data points in human feedback and synthetic comparison datasets.
- Drive strategy for aligning data collection, curation, and filtering with post-training objectives such as helpfulness, harmlessness, and faithfulness.
- Collaborate cross-functionally with engineers, alignment researchers, and product leaders to translate research into production-ready pipelines for RLHF and DPO.
- Mentor and influence junior researchers and engineers working on data-centric evaluation, reward modeling, and benchmark creation.
- Author foundational tools and metrics that connect supervision data characteristics to downstream LLM behavior and evaluation performance.
- Publish and present research that advances the field of data quality in LLM post-training, contributing to academic and industry best practices.
Requirements
- PhD or equivalent experience in machine learning, NLP, or data-centric AI, with a track record of leadership in LLM post-training or data quality research.
- 5 years of academic or industry experience post-doc
- Deep expertise in RLHF, preference data pipelines, reward modeling, or evaluation systems.
- Demonstrated experience designing and scaling data quality infrastructure — from labeling frameworks and validation metrics to automated filtering and dataset optimization.
- Strong engineering proficiency in Python, PyTorch, and ecosystem tools for large-scale training and evaluation.
- A proven ability to define, lead, and execute complex research initiatives with clear business and technical impact.
- Strong communication and collaboration skills, with experience driving strategy across research, engineering, and product teams.
Benefits
- Ownership: Equity in a fast-growing company
- Financial Wellness: 401(k) match, competitive compensation, financial coaching
- Family Support: Paid parental leave, fertility benefits, parental coaching
- Wellbeing: Medical, dental, and vision, mental health support, $500 wellness stipend
- Growth: $2,000 learning stipend, ongoing development
- Remote & Office: Stipends for home office setup, internet, commuting, and free lunch/gym in our SF office
- Time Off: Flexible PTO, 15 holidays + 2 flex days, winter #ShakeBreak where our whole office closes for a week!
- Connection: Team outings & referral bonuses
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard skills
machine learningNLPdata-centric AIRLHFreward modelingevaluation systemsPythonPyTorchdata quality infrastructuredataset optimization
Soft skills
leadershipcommunicationcollaborationmentoringinfluencingstrategic thinkingresearch initiative executioncross-functional collaborationproblem-solvingpresentation skills
Certifications
PhD