Senior AI Backend Engineer – Agent Evaluation, Quality

Salla E-Commerce Platform

Senior AI Backend Engineer responsible for evaluation systems behind multi-agent platforms. Collaborate on improving quality and development of AI agents based on system performance.

Posted 7/19/2026full-timeRemote • 🇸🇦 Saudi ArabiaSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates strong software engineering fundamentals with hands-on experience in building and evaluating LLM systems. Proficient in production-level Python or Typescript, with a focus on metrics, calibration, and automated quality assurance.

Highest-signal resume keywords

Production PythonLLM/Agent ExperienceCI/CDObservability ToolingMeasurement Mindset

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Software Engineering FundamentalsAPI DesignSystem DesignTestingLLM SystemsRegression TestingMetrics ReasoningCalibrationAdversarial TestingProduction Reliability

Tools & Technologies

LangGraphLangChainArizeLangSmith

Industry Keywords

E-commerceNLPMerchant-Facing Products

Tech Stack

Tools & technologies

PythonTypeScript

About the role

Key responsibilities & impact

Own the evaluation stack. Design and build LLM-as-judge systems, calibrate them against human labels, and make agent quality measurable per-agent and per-failure-mode.
Make the release gate real. Build per-PR eval harnesses and regression detection wired into CI, so quality is enforced automatically, not by manual passes.
Build user simulators to generate test coverage and adversarial cases before real users hit them.
Turn production signal into improvement - pipe real failures back into evaluation sets so the system compounds over time.
Partner with product to turn "what good looks like" into concrete, measurable criteria.
Grow into agent development - contribute to building and hardening the agents themselves, starting with the components you know most deeply from evaluating them.

Requirements

What you’ll need

Strong software engineering fundamentals. Production Python or Typescript (or similar), clean API and system design, testing, CI/CD. You write code others build on - evaluation infrastructure is real engineering.
Hands-on LLM/agent experience. You've built with LLMs - agents, RAG, tool/function calling, orchestration frameworks (LangGraph, LangChain, or equivalent) - and understand how they behave and break.
A measurement mindset. You reason about metrics, calibration, and experiments; you want to quantify whether something works, not just ship it.
Production experience. You've run LLM systems in production and dealt with reliability, latency, cost, and observability.
5+ years software engineering, with recent hands-on LLM/agent work.
Nice to have
Direct experience evaluating LLM/agent systems - offline/online eval, LLM-as-judge, systematic regression testing.
Observability tooling (Arize, LangSmith, or similar).
Arabic language / NLP experience.
E-commerce or merchant-facing product experience.

Benefits

Comp & perks

🌐 Worldwide ❌ Jobs You've Hidden ⭐️ Saved Jobs ✅ Applied Jobs ✉️ Email Alerts 👤 Account Salla E-Commerce Platform Website LinkedIn All Job Openings 51 - 200 employees مستقبل التجارة الإلكترونية، ابدأ تجارتك بسهولة 🛒 Senior AI Backend Engineer – Agent Evaluation, Quality 🔥 4 minutes ago 🇸🇦 Saudi Arabia – Remote ⏰ Full Time 🟠 Senior 🔧 QA Engineer (Quality Assurance) Python TypeScript Apply Now Find Hiring Managers Customize resume + cover letter Report problem ☆ Save ☑️ Mark as applied ❌ Hide 📋 Description
Own the evaluation stack. Design and build LLM-as-judge systems, calibrate them against human labels, and make agent quality measurable per-agent and per-failure-mode.
Make the release gate real. Build per-PR eval harnesses and regression detection wired into CI, so quality is enforced automatically, not by manual passes.
Build user simulators to generate test coverage and adversarial cases before real users hit them.
Turn production signal into improvement - pipe real failures back into evaluation sets so the system compounds over time.
Partner with product to turn "what good looks like" into concrete, measurable criteria.
Grow into agent development - contribute to building and hardening the agents themselves, starting with the components you know most deeply from evaluating them. 🎯 Requirements
Strong software engineering fundamentals. Production Python or Typescript (or similar), clean API and system design, testing, CI/CD. You write code others build on - evaluation infrastructure is real engineering.
Hands-on LLM/agent experience. You've built with LLMs - agents, RAG, tool/function calling, orchestration frameworks (LangGraph, LangChain, or equivalent) - and understand how they behave and break.
A measurement mindset. You reason about metrics, calibration, and experiments; you want to quantify whether something works, not just ship it.
Production experience. You've run LLM systems in production and dealt with reliability, latency, cost, and observability.
5+ years software engineering, with recent hands-on LLM/agent work.
Nice to have
Direct experience evaluating LLM/agent systems - offline/online eval, LLM-as-judge, systematic regression testing.
Observability tooling (Arize, LangSmith, or similar).
Arabic language / NLP experience.
E-commerce or merchant-facing product experience. Apply Now 📊 Check your resume score for this job Improve your chances of getting an interview by checking your resume score before you apply. Check Resume Score Similar Jobs Quality Assurance Engineer 🕒 April 1 InnovationTeam 201 - 500 🏢 Enterprise ☁️ SaaS 🔌 API Website LinkedIn All Job Openings Quality Assurance Engineer ensuring software quality and functionality at InnovationTeam. Collaborating with teams on web and mobile applications for a forward-thinking technology company. 🇸🇦 Saudi Arabia – Remote ⏰ Full Time 🟡 Mid-level 🟠 Senior 🔧 QA Engineer (Quality Assurance) Selenium SQL TFS Senior QA Engineer – Full-Stack: Backend, Mobile 🕒 February 5 Tabby 201 - 500 💳 Fintech 🛍️ eCommerce Website LinkedIn All Job Openings Senior QA Engineer designing testing strategy and performing hands-on testing for a global fintech company. Collaborating with an international engineering team for product reliability. 🇸🇦 Saudi Arabia – Remote 💰 $58M Series C on 2023-01 ⏰ Full Time 🟠 Senior 🔧 QA Engineer (Quality Assurance) Android Cloud Docker Grafana iOS Kubernetes Microservices Prometheus TypeScript Go View More QA Engineer Jobs 🌐 Worldwide Built by Lior Neu-ner. I'd love to hear your feedback — Get in touch via DM or support@remoterocketship.com Search Search Jobs by country Search jobs by city Search jobs by job title Search entry-level jobs Search junior-level jobs Search senior-level jobs Search jobs by tech stack Search jobs by contract type Search remote internships Search remote part-time jobs Remote jobs Anywhere in the World Companies Hiring Anywhere in the World Companies Hiring Sales People Anywhere in the World Companies Hiring Software Engineers Anywhere in the World Resources Advice Tips for finding remote jobs Interview questions and answers Resume examples Cover letter examples Post a job Affiliates Privacy policy Terms of service Job board SEO course AI Apply Copilot OpenClaw job finder Find jobs using your resume Jobs by Country Remote jobs anywhere in the world (Worldwide remote jobs) Remote jobs United States Remote jobs Australia Remote jobs Brazil Remote jobs Canada Remote jobs France Remote jobs Ireland Remote jobs Germany Remote jobs Netherlands Remote jobs Spain Remote jobs UK Popular Jobs Remote data analyst jobs Remote customer support jobs Remote executive assistant jobs Remote marketing jobs Remote product designer jobs Remote product manager jobs Remote project manager jobs Remote recruiter jobs Remote sales jobs Remote software engineer jobs Jobs by Type Remote full-time jobs Remote part-time jobs Remote contract jobs Remote internship jobs Remote entry-level jobs Remote jobs with no experience required Remote junior jobs (1-3 years of experience) Digital nomad jobs Remote jobs with no degree required Freelance remote jobs Temporary remote jobs Remote jobs hiring now Stay at home mom jobs