Software Developer – Agentic Evaluation

Autodesk

Software Developer creating cutting-edge AI solutions to enhance developer productivity at Autodesk. Collaborating with AI engineers and software architects on innovative projects.

Posted 6/27/2026full-timeToronto • 🇨🇦 CanadaMid-LevelSenior💰 CA$88,000 - CA$128,700 per yearWebsite

Tech Stack

Tools & technologies

PythonPyTorchScikit-Learn

About the role

Key responsibilities & impact

Develop and orchestrate multi-agent AI systems for automated test generation, test execution, and end-to-end development workflow optimization using frameworks like LangGraph, AutoGen, or the Anthropic Agent SDK (Claude Code)
Design and implement agentic workflows that coordinate multiple AI agents to autonomously drive test automation across UI, API, integration, and system levels, from test case synthesis to result evaluation, ensuring seamless integration with existing developer tools and MCP-compatible services
Build evaluation frameworks and custom benchmarks for agentic systems, including comparisons of AI agents against commercial solvers, using tools like AgentBench and Langfuse
Evaluate MCP server and tool performance across agentic pipelines, measuring latency, accuracy, context fidelity, and end-to-end task completion rates

Requirements

What you’ll need

BS/MS in Computer Science, Machine Learning, or a related applied AI field
Expertise in Python and ML frameworks (PyTorch, Transformers, scikit-learn)
Experience with Large Language Models applied to software understanding or test generation
Knowledge of AI evaluation methodologies and metrics for agentic task completion and test quality
Strong foundation in statistical analysis and experimental design
Experience with developer workflow and productivity measurement frameworks

Benefits

Comp & perks

Health insurance
401(k) matching
Flexible work hours
Paid time off
Professional development opportunities

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

PythonMachine LearningLangGraphAutoGenAnthropic Agent SDKPyTorchTransformersScikit-learnStatistical AnalysisExperimental Design

Certifications

BS in Computer ScienceMS in Computer ScienceBS in Machine LearningMS in Machine Learning