FREE ACCESS
5,000–10,000 jobs/day
See all jobs on JobTailor
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Core Competencies
Role fitCore Competencies
Use this summary to align your resume positioning with the role.
Demonstrates expertise in designing evaluation methodologies and benchmarks for API-based coding agents, with a strong foundation in experimental design and statistical analysis. Proficient in Python and/or Go, with a deep understanding of modern LLMs and AI evaluation methodologies.
Highest-signal resume keywords
PhD In Computer ScienceExperimental DesignStatistical AnalysisProgramming In PythonAPI-Based Foundation Models
ATS Keywords
Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills
Statistical AnalysisExperimental DesignProgramming In PythonProgramming In GoBenchmark DesignDataset CreationEvaluation FrameworksLarge-Scale Data AnalysisIndependent ResearchTechnical Reporting
Soft Skills
CommunicationCollaboration
Tools & Technologies
Research InfrastructureSimulatorsDeveloper ToolsCoding Assistants
Industry Keywords
Machine LearningAI EvaluationInformation RetrievalSoftware EngineeringAgent Architectures
Tech Stack
Tools & technologiesGoPython
About the role
Key responsibilities & impact- Design rigorous evaluation methodologies for API-based coding agents, with and without access to external tools.
- Build statistically sound benchmarks that measure cost, latency, accuracy, reliability, and developer productivity.
- Develop novel techniques for optimizing agent context, retrieval, memory, and tool interactions while preserving correctness.
- Design and analyze large-scale experimental campaigns using production APIs across multiple foundation models.
- Build reusable research infrastructure, datasets, simulators, and evaluation frameworks for agentic systems.
- Publish technical reports and contribute research findings that influence both product direction and the broader AI community.
- Collaborate closely with engineering teams to translate research into production-ready optimization systems.
- Stay at the forefront of advances in LLMs, agent architectures, retrieval systems, and AI evaluation methodologies.
Requirements
What you’ll need- PhD or equivalent research experience in Computer Science, Machine Learning, Mathematics, Statistics, or a related quantitative field.
- Strong background in experimental design, statistical analysis, and empirical evaluation.
- Excellent programming skills in Python and/or Go, with experience building research infrastructure.
- Demonstrated ability to conduct independent research and communicate findings through publications, technical reports, or open-source projects.
- Deep understanding of modern LLMs, retrieval systems, or AI agents.
- Experience working with API-based foundation models such as Claude, GPT, Gemini, or similar systems.
- Publications in machine learning, systems, information retrieval, software engineering, or AI evaluation.
- Experience designing benchmarks, datasets, or evaluation frameworks.
- Familiarity with developer tools, coding assistants, or software engineering workflows.
- Experience with distributed experimentation and large-scale data analysis.
Benefits
Comp & perks- Equal Opportunity Statement
- Celebration of diversity
- Commitment to an inclusive environment