Data Annotation Specialist – Generalist

Cohere

Data Annotation Specialist reviewing, assessing, and improving AI model outputs in a remote contract role. Contributing to high-impact annotation projects for a security-focused AI company.

Posted 7/24/2026contractRemote • 🇨🇦 CanadaJunior💰 CA$30 per hourWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in AI data annotation and LLM evaluation, with a strong focus on quality assurance and preference ranking. Capable of adapting to evolving tasks while maintaining high standards of accuracy and detail in multimodal data annotation.

Highest-signal resume keywords

AI Data AnnotationLLM EvaluationQuality AssuranceAttention to DetailWritten English Proficiency

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Data AnnotationPreference RankingModel EvaluationError DetectionPrompt Authoring

Soft Skills

Contextual JudgmentComfort with AmbiguityStrong Reading ComprehensionTime ManagementIndependent Work

Tools & Technologies

JSONCSV/TSVMarkdownXMLYAML

Industry Keywords

Model Performance ReportingMultimodal Data AnnotationCalibration ExercisesInter-Annotator AgreementAdversarial Testing

About the role

Key responsibilities & impact

Evaluate and rank model outputs: Complete preference and comparison tasks, assessing which responses best conform to project guidelines for accuracy, helpfulness, tone, and safety, and writing clear justifications for your judgments.
Stress-test and break models: Probe models adversarially to surface failure modes, unsafe behavior, and capability gaps, and document reproducible cases that engineering and research teams can act on.
Create datasets: Author high-quality prompts, responses, and exemplars to build training and evaluation datasets, following detailed specifications and editing machine-written or human-written outputs to standard.
Build and apply rubrics and taxonomies: Contribute to the design of grading criteria and rubrics, then apply them consistently to produce structured, high-quality annotations across task types.
Annotate and correct multimodal data: Label, audit, and rectify inaccuracies across text, image, and structured data, maintaining a high standard of data integrity and accuracy.
Calibrate and maintain consistency: Participate in calibration exercises and inter-annotator agreement checks to align on standards, and flag ambiguous or uncovered edge cases rather than guessing, since a single misjudgment replicated at scale degrades a model.
Adapt to experimental work: Take on new and evolving task types as project needs shift, applying sound judgment in areas where guidelines are still being developed.
Report on model performance: Surface and communicate quality and performance trends in model and agent behavior, giving cross-functional partners clear, well-evidenced feedback on where models succeed, fail, and degrade.

Requirements

What you’ll need

1+ years of experience in AI data annotation, LLM evaluation, content moderation, research, or a related analytical role, with exposure to quality assurance, and/or preference ranking.
Experience applying detailed guidelines to complex and often ambiguous content, with strong contextual and sociocultural judgment, sensitivity to nuance, tone, and register, and the ability to reason well in cases where there is no single correct answer.
Comfort with ambiguity: a willingness to flag unclear or uncovered edge cases rather than guess, and to work productively on novel, experimental tasks whose definitions are still evolving.
A sharp, curious eye for inconsistencies, subtle errors, and model failure modes, including the instinct to probe models adversarially and surface where they break. Familiarity with how large language models behave, such as hallucination, sycophancy, and instruction-following gaps, is a plus.
Excellent command of written English and strong reading comprehension, with the ability to clearly justify your evaluations, including why an output is correct or incorrect, high-quality or low-quality, and to write clear prompts and exemplars that explain your reasoning. Bonus points if you are fluent in another language!
Strong attention to detail and commitment to accuracy, with the ability to maintain consistency across high-volume and monotonous tasks.
Comfort working with annotation platforms and structured formats such as JSON, CSV/TSV, Markdown, XML, and YAML.
Strong execution in a remote environment, including good time management, comfort using new tools, and the ability to work independently in a global, asynchronous team.

Benefits

Comp & perks

Bring Your Own Device (BYOD) 💻