Design and develop automated adversarial testing methodologies — including single-turn, multi-turn, and multimodal attack strategies — to proactively identify vulnerabilities in Pinterest's Generative AI products.
Build and calibrate hybrid evaluation pipelines combining LLM-based judges, classifiers, and rule-based systems to accurately detect safety violations, policy breaches, bias, and representational harms.
Develop and operationalize harm taxonomies grounded in industry standards and Pinterest's Responsible AI and Trust & Safety threat models.
Design adaptive refinement loops that learn from attack outcomes (near-misses, partial failures) to iteratively surface deeper and previously unknown vulnerabilities.
Bring scientific rigor and statistical methods to the evaluation of AI safety — including benchmark dataset construction, evaluation calibration, and success-metric definition (vulnerability severity, coverage breadth, pre-launch risk reduction).
Work cross-functionally to build relationships, proactively communicate key findings, and collaborate closely with ML engineers, Trust & Safety specialists, policy teams, product managers, and legal partners to ensure safe product launches.
Relentlessly focus on impact, whether through influencing product safety strategy, advancing responsible AI metrics, or improving critical evaluation processes.
Mentor and up-level junior data scientists and cross-functional partners on adversarial evaluation, responsible AI methodologies, and safety-aware data science practices.

Requirements

5+ years of experience analyzing data in a fast-paced, data-driven environment with proven ability to apply scientific methods to solve real-world problems on web-scale data.
Strong interest and hands-on experience in one or more of: AI safety, adversarial machine learning, red teaming, responsible AI, or trust & safety.
Deep familiarity with large language models (LLMs), generative AI systems, and their failure modes — including prompt injection, jailbreaks, bias, and safety violations.
Experience designing and calibrating evaluation frameworks for AI systems — including LLM-as-judge, classifier-based evaluation, and benchmark dataset construction.
Strong quantitative programming (Python) and data manipulation skills (SQL/Spark); experience with ML pipelines and large-scale experimentation.
Familiarity with AI safety taxonomies and frameworks (e.g., OWASP LLM Top 10, MITRE ATLAS) is strongly preferred.
Ability to work independently, drive ambiguous projects end-to-end, and operate with high ownership.
Excellent written and verbal communication skills, with the ability to explain complex technical findings to both technical and non-technical partners.
A team player eager to partner across Responsible AI, Trust & Safety, Product, Engineering, Policy, and Legal to turn safety insights into action.

Benefits

Information regarding the culture at Pinterest and benefits available for this position can be found here.

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

adversarial testing methodologieslarge language models (LLMs)generative AI systemsevaluation frameworksquantitative programming (Python)data manipulation (SQL/Spark)ML pipelinesbenchmark dataset constructionstatistical methodsevaluation calibration

Soft Skills

communication skillscollaborationmentoringindependenceownershipinfluencefocus on impactrelationship buildingproblem-solvingteam player