QA Lead

InnoData

QA/Evaluation Lead at Innodata managing evaluation frameworks and quality assurance for AI/ML programs. Designing metrics and frameworks to ensure data service reliability in federal projects.

Posted 7/27/2026full-timeWashington D.C. • Washington • 🇺🇸 United StatesSenior💰 $45 - $50 per hourWebsite

Tech Stack

Tools & technologies

PythonTypeScript

About the role

Key responsibilities & impact

Design and own the inter-annotator agreement (IAA) methodology for the Phase 1 demonstration corpus — metric selection (Cohen's kappa, Fleiss, Krippendorff's alpha), sampling design, adjudication workflow, and agreement thresholds
Define evaluation framework architecture: test and evaluation plans, IAA targets, drift detection gates, and model performance metrics per SOW Section 2.9
Configure and operate sampling-based quality control across the self-service and white-glove annotation paths during Phase D corpus production
Design and implement confidence-threshold escalation routing from automated annotation to senior-annotator adjudication
Validate quality scoring and IAA computation within the Innodata data layer
Support AI Solutions Engineer on evaluation design for SAM 2 and Frontier model API validation — define what 'good enough' looks like quantitatively
Produce evaluation framework documentation for the Phase 1 NPP closeout package, including per-DataCard documentation with the SA

Requirements

What you’ll need

Bachelor's degree in Statistics, Data Science, Computer Science, or related quantitative field required; Master's degree preferred. Equivalent experience may substitute for degree on a 2-for-1 basis.
6+ years total professional experience, 4+ years in data quality, evaluation methodology, or QA on AI/ML programs
IAA methodology expertise — Cohen's kappa, Fleiss' kappa, Krippendorff's alpha: hands-on, not theoretical
Evaluation framework design for AI/ML training data programs
QC process design: sampling methodology, escalation workflows, adjudication protocols
Python for QC tooling, metric computation, and statistical analysis
Active Secret clearance with TS/SCI eligibility.

Benefits

Comp & perks

🌐 Worldwide ❌ Jobs You've Hidden ⭐️ Saved Jobs ✅ Applied Jobs ✉️ Email Alerts 👤 Account InnoData Website LinkedIn All Job Openings 2 - 10 employees Founded 2019 🤝 B2B 💼 Consulting 🌍 Social Impact B2B
Consulting
Social Impact InnoData is INNOvation DATA SCS, an Italian social cooperative based in Foggia that identifies itself as a provider of technological solutions. The company website (currently under maintenance) highlights "Soluzioni tecnologiche" (technological solutions) and emphasizes social impact ("Impatto sociale"). Contact details listed include Via Francesco Crispi 65, 71121 Foggia, Italy. Based on the available information, InnoData appears to operate at the intersection of technology and social impact, likely offering tech-focused services to other organizations. QA Lead Job not on LinkedIn 🔥 6 minutes ago 🏢🏡 Washington D.C. – Hybrid 💵 $45 - $50 / hour ⏰ Full Time 🟠 Senior 🔧 QA Engineer (Quality Assurance) Python TypeScript Apply Now Find Hiring Managers Customize resume + cover letter Report problem ☆ Save ☑️ Mark as applied ❌ Hide 📋 Description
Design and own the inter-annotator agreement (IAA) methodology for the Phase 1 demonstration corpus — metric selection (Cohen's kappa, Fleiss, Krippendorff's alpha), sampling design, adjudication workflow, and agreement thresholds
Define evaluation framework architecture: test and evaluation plans, IAA targets, drift detection gates, and model performance metrics per SOW Section 2.9
Configure and operate sampling-based quality control across the self-service and white-glove annotation paths during Phase D corpus production
Design and implement confidence-threshold escalation routing from automated annotation to senior-annotator adjudication
Validate quality scoring and IAA computation within the Innodata data layer
Support AI Solutions Engineer on evaluation design for SAM 2 and Frontier model API validation — define what 'good enough' looks like quantitatively
Produce evaluation framework documentation for the Phase 1 NPP closeout package, including per-DataCard documentation with the SA 🎯 Requirements
Bachelor's degree in Statistics, Data Science, Computer Science, or related quantitative field required; Master's degree preferred. Equivalent experience may substitute for degree on a 2-for-1 basis.
6+ years total professional experience, 4+ years in data quality, evaluation methodology, or QA on AI/ML programs
IAA methodology expertise — Cohen's kappa, Fleiss' kappa, Krippendorff's alpha: hands-on, not theoretical
Evaluation framework design for AI/ML training data programs
QC process design: sampling methodology, escalation workflows, adjudication protocols
Python for QC tooling, metric computation, and statistical analysis
Active Secret clearance with TS/SCI eligibility. Apply Now 📊 Check your resume score for this job Improve your chances of getting an interview by checking your resume score before you apply. Check Resume Score 🌐 Worldwide Built by Lior Neu-ner. I'd love to hear your feedback — Get in touch via DM or support@remoterocketship.com Search Search Jobs by country Search jobs by city Search jobs by job title Search entry-level jobs Search junior-level jobs Search senior-level jobs Search jobs by tech stack Search jobs by contract type Search remote internships Search remote part-time jobs Remote jobs Anywhere in the World Companies Hiring Anywhere in the World Companies Hiring Sales People Anywhere in the World Companies Hiring Software Engineers Anywhere in the World Resources Advice Tips for finding remote jobs Interview questions and answers Resume examples Cover letter examples Post a job Affiliates Is Remote Rocketship legit? Privacy policy Terms of service Job board SEO course OpenClaw job finder Find jobs using your resume Jobs by Country Remote jobs anywhere in the world (Worldwide remote jobs) Remote jobs United States Remote jobs Australia Remote jobs Brazil Remote jobs Canada Remote jobs France Remote jobs Ireland Remote jobs Germany Remote jobs Netherlands Remote jobs Spain Remote jobs UK Popular Jobs Remote data analyst jobs Remote customer support jobs Remote executive assistant jobs Remote marketing jobs Remote product designer jobs Remote product manager jobs Remote project manager jobs Remote recruiter jobs Remote sales jobs Remote software engineer jobs Jobs by Type Remote full-time jobs Remote part-time jobs Remote contract jobs Remote internship jobs Remote entry-level jobs Remote jobs with no experience required Remote junior jobs (1-3 years of experience) Digital nomad jobs Remote jobs with no degree required Freelance remote jobs Temporary remote jobs Remote jobs hiring now Stay at home mom jobs