Product Operations Specialist
Unify
SQL
Annotation QA Analyst, Content Platform
Spotify
full-time
Posted on:
Origin: • 🇺🇸 United States • New York
Visit company websiteSalary
💰 $82,921 - $118,459 per year
Job Level
Mid-LevelSenior
Tech Stack
SQL
About the role
- Proactively review and ensure annotated data is in line with Spotify’s standards and policies, including running quality reviews prioritized by business needs.
- Ensure high-quality and timely results for R&D collaborator teams (Product and Engineering) via established quality framework / quality assurance processes, using metrics such as agreement rates and consensus.
- Run QA expert reviews on the most difficult edge cases in projects in order to address and resolve ambiguity.
- Surface themes, suggestions, and feedback for the annotation team per project. Communicate complex concepts and findings in a clear and effective manner to non-technical and technical audiences.
- Contribute to feedback loops between annotation teams, R&D collaborator teams (Product and Engineering), content policy experts and/or Trust & Safety policy experts, in order to continuously iterate on data analysis workflows.
- Collaborate with R&D collaborator teams (Product and Engineering) to ensure proper project guidelines exist per use case prior to annotator training (e.g. ensuring proper in scope vs out of scope examples are generated). If necessary, update or revise project guidelines based on results from initial rounds of annotations.
- Proactive collaborate with other members of the Annotation Platform Ops team across all in-flight projects.
Requirements
- Well-versed in annotation, data analysis, quality assurance, and human-in-the-loop best practices.
- Familiar with AI / LLM-driven workflows.
- Comfortable with large-scale review and contributing to very large datasets across multi-modal content types (e.g., text, audio, images, video).
- Passionate about catalog safety and moderation (e.g., Trust & Safety) and a desire to apply QA frameworks across diverse domains.
- Self-starter who can collaborate with cross-functional partners in a dynamic environment and deliver results.
- Excellent verbal and written communication skills.
- Experience working outside of the safety and moderation space is a plus.
- Good grasp of machine learning lifecycles from goal setting to data collection/labeling to model development and deployment is a plus.
- Basic knowledge of SQL is a plus.
