GoFasti

AI Engineer – Evaluation Systems

GoFasti

contract

Posted on:

Location Type: Remote

Location: Anywhere in Latin America

Visit company website

Explore more

✨ AI Apply
Apply

Salary

💰 $4,500 - $5,000 per month

Tech Stack

AWSDockerPostgresPython

About the role

  • Design a structured, configurable evaluation engine
  • Combine deterministic checks with LLM-as-judge verdicts
  • Build calibration workflows using expert-labeled examples
  • Measure precision and recall properly (not raw agreement)
  • Handle delayed outcomes and low-confidence review flows
  • Store structured verdicts that power dashboards and analytics
  • Monitor drift and trigger recalibration when alignment drops
  • Specialized judges for specific failure patterns
  • Content-level evaluation (with careful PII handling)
  • Error analysis pipelines that drive workflow redesign
  • A/B testing for judge versions
  • Eval-driven fine-tuning data curation

Requirements

  • 4+ years total backend / ML engineering experience
  • 2+ years building production AI/LLM systems
  • Experience with Python, Docker and PostgreSQL.
  • Experience with AWS, OpenAI, Anthropic, and other LLM APIs
  • Proven experience building LLM-based systems in production environments where output quality and reliability were critical.
  • Experience developing evaluation, QA, or scoring pipelines to assess model performance.
  • Strong understanding of precision/recall trade-offs and working with real-world data.
  • Ability to design and implement systems that produce reliable, structured outputs from LLMs.
  • Strong Python programming skills, including experience with asynchronous programming.
  • Experience deploying and managing services on AWS.
Benefits
  • The talent will work REMOTELY allocated at our client.
Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools
backend engineeringmachine learning engineeringPythonDockerPostgreSQLAWSOpenAIAnthropicA/B testingevaluation pipelines
Soft Skills
problem-solvinganalytical thinkingattention to detailcommunicationcollaborationadaptabilitycritical thinkingtime managementcreativityorganizational skills