AI Evaluation Engineer – Software Engineering Domain

Gramian Consulting

AI Evaluation Engineer focused on designing tasks to evaluate AI systems in software engineering domains. Contributing expertise in debugging, workflows, and infrastructure-heavy environments.

Posted 5/27/2026contractRemote • 🇪🇬 EgyptMid-LevelSeniorWebsite

About the role

Key responsibilities & impact

Design realistic terminal-based benchmark tasks for AI evaluation systems
Create technically deep debugging and investigation scenarios
Develop task specifications involving infrastructure, workflows, pipelines, or operational failures
Write clear solution approaches and deterministic evaluation criteria
Identify realistic edge cases, failure modes, and system constraints
Design multi-step reasoning challenges across complex technical environments
Contribute expertise across one or more engineering or operational domains
Review and refine benchmark quality, difficulty, and validation logic
Collaborate with reviewers and researchers on AI evaluation workflows

Requirements

What you’ll need

3–10 years of experience in software engineering or related technical domains
Strong debugging, analytical, and systems reasoning skills
Good understanding of system architecture, dependencies, and operational processes
Experience with terminal, CLI, automation, or developer tooling workflows
Exposure to AI systems, LLMs, benchmarking, or evaluation frameworks is preferred
Ability to design technically rigorous and realistic engineering scenarios

Benefits

Comp & perks

🌐 Worldwide ❌ Jobs You've Hidden ⭐️ Saved Jobs ✅ Applied Jobs ✉️ Email Alerts 👤 Account Gramian Consulting Website LinkedIn All Job Openings 2 - 10 employees Founded 2025 🎯 Recruiter 🤖 Artificial Intelligence 📚 Education Recruitment
Artificial Intelligence
Education Gramian Consulting is a remote-first consulting firm that connects engineering and data/AI talent with organizations through talent augmentation, recruiting, dedicated teams, and contractor management. The firm provides Data & AI services including LLM training and fine-tuning, AI agents and assistants, MLOps, and AI infrastructure, and it offers mentorship and education programs for career readiness, interview preparation, and international market orientation. Rooted in hands-on engineering and recruiting experience, Gramian helps clients scale technical teams and extract business value from AI while developing individual talent. AI Evaluation Engineer – Software Engineering Domain 🔥 0 minutes ago 🇪🇬 Egypt – Remote ⏳ Contract/Temporary 🟡 Mid-level 🟠 Senior 🧑‍💻 Full-stack Engineer Apply Now Find Hiring Managers Customize resume + cover letter Report problem ☆ Save ☑️ Mark as applied ❌ Hide 📋 Description
Design realistic terminal-based benchmark tasks for AI evaluation systems
Create technically deep debugging and investigation scenarios
Develop task specifications involving infrastructure, workflows, pipelines, or operational failures
Write clear solution approaches and deterministic evaluation criteria
Identify realistic edge cases, failure modes, and system constraints
Design multi-step reasoning challenges across complex technical environments
Contribute expertise across one or more engineering or operational domains
Review and refine benchmark quality, difficulty, and validation logic
Collaborate with reviewers and researchers on AI evaluation workflows 🎯 Requirements
3–10 years of experience in software engineering or related technical domains
Strong debugging, analytical, and systems reasoning skills
Good understanding of system architecture, dependencies, and operational processes
Experience with terminal, CLI, automation, or developer tooling workflows
Exposure to AI systems, LLMs, benchmarking, or evaluation frameworks is preferred
Ability to design technically rigorous and realistic engineering scenarios Apply Now 📊 Check your resume score for this job Improve your chances of getting an interview by checking your resume score before you apply. Check Resume Score Similar Jobs WordPress Software Developer 🕒 2 days ago Scale Army Careers 11 - 50 🎯 Recruiter 🤝 B2B Website LinkedIn All Job Openings WordPress Software Developer for a digital-focused company. Building and optimizing high-performing WordPress websites for diverse clients across various industries. 🇪🇬 Egypt – Remote 💵 $1.8k - $2.2k / month ⏳ Contract/Temporary 🟢 Junior 🟡 Mid-level 🧑‍💻 Full-stack Engineer DNS JavaScript jQuery Vue.js WordPress Full-Stack Developer, AI Trainer 🕒 2 days ago Anyone AI 11 - 50 📚 Education 🤖 Artificial Intelligence 🎯 Recruiter Website LinkedIn All Job Openings Full-Stack Developer involved in AI project for Anyone AI. Responsibilities include coding tasks, peer reviews, and testing. 🇪🇬 Egypt – Remote 💵 $15 - $35 / hour 💰 $1.5M Pre Seed Round - Anyone AI on 2022-06 ⏳ Contract/Temporary 🟡 Mid-level 🟠 Senior 🧑‍💻 Full-stack Engineer Java JavaScript Jest JUnit Python TypeScript Go Mid-Level Software Engineer 🕒 May 19 Scale Army Careers 11 - 50 🎯 Recruiter 🤝 B2B Website LinkedIn All Job Openings Mid-Level Software Engineer at a fintech SaaS company developing scalable lending technology and borrower-facing workflows. 🇪🇬 Egypt – Remote 💵 $2.5k - $3k / month ⏳ Contract/Temporary 🟡 Mid-level 🟠 Senior 🧑‍💻 Full-stack Engineer AWS Docker EC2 Postgres SQL View More Full-stack Engineer Jobs 🌐 Worldwide Built by Lior Neu-ner. I'd love to hear your feedback — Get in touch via DM or support@remoterocketship.com Search Search Jobs by country Search jobs by city Search jobs by job title Search entry-level jobs Search junior-level jobs Search senior-level jobs Search jobs by tech stack Search jobs by contract type Search remote internships Search remote part-time jobs Remote jobs Anywhere in the World Companies Hiring Anywhere in the World Companies Hiring Sales People Anywhere in the World Companies Hiring Software Engineers Anywhere in the World Resources Advice Tips for finding remote jobs Interview questions and answers Resume examples Cover letter examples Post a job Affiliates Privacy policy Terms of service Job board SEO course AI Apply Copilot OpenClaw job finder Jobs by Country Remote jobs anywhere in the world (Worldwide remote jobs) Remote jobs United States Remote jobs Australia Remote jobs Brazil Remote jobs Canada Remote jobs France Remote jobs Ireland Remote jobs Germany Remote jobs Netherlands Remote jobs Spain Remote jobs UK Popular Jobs Remote data analyst jobs Remote customer support jobs Remote executive assistant jobs Remote marketing jobs Remote product designer jobs Remote product manager jobs Remote project manager jobs Remote recruiter jobs Remote sales jobs Remote software engineer jobs Jobs by Type Remote full-time jobs Remote part-time jobs Remote contract jobs Remote internship jobs Remote entry-level jobs Remote jobs with no experience required Remote junior jobs (1-3 years of experience) Digital nomad jobs Remote jobs with no degree required Freelance remote jobs Temporary remote jobs Remote jobs hiring now Stay at home mom jobs

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

debugginganalytical skillssystems reasoningsystem architectureautomation workflowsdeveloper toolingAI systemsLLMsbenchmarkingevaluation frameworks

Soft Skills

collaborationcommunication