AI Evaluation Engineer

GovWorx

AI Evaluation Engineer at GovWorx improving AI systems for public safety agencies. Responsible for building automated evaluation pipelines, analyzing AI model performance, and collaborating with cross-functional teams.

Posted 7/17/2026full-timeRemote • Colorado • 🇺🇸 United StatesMid-LevelSenior💰 $120,000 - $170,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and maintaining automated AI evaluation pipelines, with strong capabilities in Python and SQL for analyzing AI performance. Proficient in prompt engineering, statistical methods, and developing evaluation metrics to ensure continuous improvement of AI systems.

Highest-signal resume keywords

Python DevelopmentSQL AnalysisPrompt EngineeringStatistical MethodsProduction LLM Applications

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Automated AI Evaluation PipelinesEvaluation Metrics DesignStatistical TechniquesA/B TestingRegression Testing FrameworksData AnalysisExperiment DesignError AnalysisQuantitative Evaluation MethodsDashboard Development

Certifications & Qualifications

US CitizenshipFBI Fingerprint Check

Industry Keywords

Responsible AIGenerative AIModel PerformanceContinuous Model ImprovementAI Quality Metrics

Tech Stack

Tools & technologies

PythonSQL

About the role

Key responsibilities & impact

Design, build, and maintain automated AI evaluation pipelines for production LLM applications
Develop prompt engineering strategies and iterate on prompts and compare LLMs using quantitative evaluation methods
Build offline evaluation datasets and regression testing frameworks to measure AI performance over time
Analyze production AI behavior using Python, SQL, and statistical techniques to identify opportunities for improvement
Design experiments, A/B tests, and benchmarking methodologies for evaluating prompt and model changes
Develop dashboards and reporting that communicate AI quality, reliability, and performance metrics
Partner with engineering and product teams to safely deploy and monitor improvements to production AI systems
Investigate model failures through detailed error analysis and recommend improvements to prompts, evaluation datasets, and workflows
Help establish best practices for Responsible AI, evaluation methodologies, and continuous model improvement

Requirements

What you’ll need

Must have US citizenship and pass FBI fingerprint and background check in multiple states
3+ years of experience in software engineering, machine learning, data science, or a related technical field
Experience designing evaluation metrics and interpreting AI model performance
Understanding of statistical methods including hypothesis testing and experiment design
Strong Python development experience
Strong SQL skills with experience analyzing large datasets
Experience building or supporting production LLM or Generative AI applications
Experience with prompt engineering and systematic prompt evaluation

Benefits

Comp & perks

Offers Bonus
Benefits (Health, Dental, Vision)
Flexible Time Off