AI Query Evaluation Specialist – Copilot Competitive Intelligence

BPCS, Comprehensive marketing solutions, ltd.

AI Query Evaluation Specialist evaluating AI-powered search and assistant experiences for Blueprint Technologies. Supporting initiatives focused on improving quality of datasets and benchmarks for AI systems.

Posted 4/29/2026full-timeRemote • Washington • 🇺🇸 United StatesMid-LevelSenior💰 $80,000 - $95,000 per yearWebsite

About the role

Key responsibilities & impact

Review and analyze real user query logs to identify queries with clear intent and strong representativeness of English-speaking markets
Curate and maintain high-quality datasets used to evaluate AI systems such as Microsoft Copilot, ChatGPT, and Gemini
Annotate queries across multiple evaluation dimensions, including:
Need for web search or external information retrieval
Presence or likelihood of personally identifiable information (PII)
Requirement for domain-specific or professional expertise
Additional attributes relevant to AI response quality
Ensure annotations are consistent, structured, and aligned with evolving evaluation guidelines
Use tools such as Excel to organize, review, and summarize evaluation outputs
Identify trends in query patterns and provide feedback to improve dataset coverage and quality
Maintain a high level of attention to detail, documentation quality, and evaluation integrity

Requirements

What you’ll need

Strong English reading comprehension with the ability to interpret subtle differences in user intent
Demonstrated analytical thinking and logical reasoning skills
Experience working with structured data or annotation workflows
Familiarity with tools such as Microsoft Excel or similar data analysis tools
Strong user empathy and understanding of how diverse users formulate queries
Curiosity and familiarity with modern AI tools (e.g., Copilot, ChatGPT, Gemini)
High attention to detail with a track record of delivering consistent, high-quality work
Reliable, proactive, and adaptable in fast-changing environments
Prior experience in data annotation, content evaluation, or dataset curation for AI or search products (preferred)
Experience with AI evaluation, search relevance, or linguistic analysis (preferred)
Basic statistical or data analysis knowledge (preferred)
Demonstrated ability to quickly learn and interpret unfamiliar domains (preferred)
Fluency in English plus at least one additional language: Japanese, Korean, French, Chinese, German, or Italian (preferred)

Benefits

Comp & perks

Medical, dental, and vision coverage
Flexible Spending Account
401k program
Competitive PTO offerings
Parental Leave
Opportunities for professional growth and development

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

data annotationdata analysisstatistical analysisAI evaluationlinguistic analysisdataset curationquery analysisinformation retrievalstructured data managementevaluation guidelines

Soft Skills

analytical thinkinglogical reasoninguser empathyattention to detailcuriosityadaptabilityproactivitycommunicationinterpretation of user intentdocumentation quality