User Researcher, AI Evaluations

Notion

User Researcher assessing the quality of AI-powered experiences at Notion. Defining evaluation frameworks and conducting user studies for product improvement in a collaborative workspace.

Posted 6/19/2026full-timeSan Francisco • California, New York • 🇺🇸 United StatesMid-LevelSenior💰 $196,000 - $230,000 per yearWebsite

Tech Stack

Tools & technologies

PythonSQL

About the role

Key responsibilities & impact

Define what “good” looks like (frameworks & rubrics)
Run recurring evals (longitudinal & feature-specific)
Anchor evaluation in real workflows (context > isolated feedback)
Identify failure modes & recovery behavior (guardrails)
Operationalize evaluation with partners (process & tooling)

Requirements

What you’ll need

5+ years doing UX research in industry
Ability to operationalize insight into measurement
AI fluency and systems thinking
Clear communication and impact orientation
Strong UX research craft (quant + qual)
Pragmatism in fast-moving environments
Familiarity with LLM-as-judge methods, prompt design for evaluators, or "golden dataset" creation
Experience using AI research tooling for rapid synthesis and communication (e.g., Dovetail, Listen Labs, Maze, Outset, etc.)
Experience using data querying languages (e.g., SQL), scripting languages (e.g., Python), or statistical/mathematical software (e.g., R, SAS, Matlab, etc.)
Master’s or PhD in HCI, Psychology, Behavioral Science, Anthropology, Sociology, or a related field

Benefits

Comp & perks

highly competitive cash compensation
equity
benefits

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

UX researchquantitative researchqualitative researchdata querying languagesSQLscripting languagesPythonstatistical softwareRSAS

Soft Skills

clear communicationimpact orientationpragmatismsystems thinkingoperationalization of insight

Certifications

Master’s in HCIPhD in HCIMaster’s in PsychologyPhD in PsychologyMaster’s in Behavioral SciencePhD in Behavioral ScienceMaster’s in AnthropologyPhD in AnthropologyMaster’s in SociologyPhD in Sociology