Own agent quality end-to-end: diagnosis, improvement, and validation across SmartAssist's orchestrator and subagents
Identify failure modes across quality dimensions factual accuracy, completeness, tone, actionability, and latency and prioritize what to fix
Drive quality improvements through prompt engineering, context engineering, and RAG retrieval tuning
Extend and mature our evaluation framework: scorers, golden datasets, regression gates, and online evaluation for production traffic
Close the feedback loop ensure that every change has a measurable, attributable quality signal
Collaborate with our Agent Architecture lead to distinguish quality problems that require prompt/context solutions from those that require structural fixes
Establish repeatable methodology that scales beyond any single agent or subagent

Requirements

8+ years of software engineering experience, with at least 2 years working directly with LLMs in production
Deep, hands-on experience with prompt engineering and context engineering, you understand how model behavior changes with framing, structure, and input design
Strong working knowledge of RAG architectures: chunking strategies, embedding models, retrieval evaluation, and failure diagnosis
Experience building or extending LLM evaluation frameworks, you have designed scorers, worked with golden datasets, and thought carefully about what good looks like
Fluency in agent system design, you don't need to own the architecture, but you can engage as a peer on architectural tradeoffs that affect quality
Strong Python skills; comfortable working in data-heavy environments (Databricks, Delta tables, or equivalent)
Ability to communicate complex quality findings (written and verbal) to both technical and non-technical stakeholders, you can explain what’s broke, why it matters, and what needs to happen next without losing the room
Strong cross-functional judgment, you know when to escalate, when to resolve independently, and how to build credibility across engineering, product, and AI platform teams
A bias for clarity in ambiguous situations, when failure modes are murky and trade-offs are real, you bring structure and a clear point of view rather than waiting for consensus
Legally eligible to work in the U.S. on an ongoing basis
BS or MS in Computer Science, a related field, or equivalent industry experience.

Benefits

Employer subsidized medical/vision and dental coverage for full-time employees
401k Match to help you save for your future (50% of your contribution up to the first 6% of your eligible pay)
Monthly stipend to support your work and productivity
Flexible Time Away Program, plus Sick Time Off
US employees are automatically covered under Smartsheet-sponsored life insurance, short-term, and long-term disability plans
US employees receive 12 paid holidays per year
Up to 24 weeks of Parental Leave
Personal paid Volunteer Day to support our community
Opportunities for professional growth and development including access to Udemy online courses
Company Funded Perks, including a counseling membership, local retail discounts, and your own personal Smartsheet account
Teleworking options from any registered location in the U.S. (role specific)

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

software engineeringprompt engineeringcontext engineeringRAG architecturesembedding modelsretrieval evaluationLLM evaluation frameworksPythondata-heavy environmentsfailure diagnosis

Soft Skills

communicationcross-functional judgmentclarity in ambiguous situationscollaborationproblem-solvingstakeholder engagementindependent resolutioncredibility buildingfeedback loop managementmethodology establishment

Certifications

BS in Computer ScienceMS in Computer Science