Senior LLM Systems Engineer

Risk Labs Foundation

Senior LLM Systems Engineer improving accuracy and resilience of LLM-driven oracle automation systems. Collaborating on evaluations, tooling, and debugging for production operations.

Posted 5/25/2026full-timeRemote • 🌎 Anywhere in the WorldSenior💰 $100,000 - $200,000 per yearWebsite

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

PythonTypeScriptLLMsmodel selectionprompt engineeringstructured outputsregression testsquality metricsdebuggingtool usage

Soft Skills

strong ownershipclear written communicationhigh agencyreasoning in uncertain environmentsdebugging ability

Tools & Technologies

APIsdatabasesqueueslogsdashboards

Industry Keywords

oracle automationevaluation coveragemodel qualityadversarial inputsfeedback loops

Tech Stack

Tools & technologies

OraclePythonTypeScript

About the role

Key responsibilities & impact

We are hiring a Senior LLM Systems Engineer to own and improve the LLM-driven components of our oracle automation stack.
Focus on the accuracy, performance, resilience, and operational quality of the systems that use models to reason about wide ranging prediction market rules, evidence, and oracle outcomes.
Build the evaluations, observability, tooling, fallbacks, and feedback loops that make LLM behavior measurable and dependable in real-world conditions.
Improve prompts, model selection, tool usage, structured outputs, retrieval, and evaluation coverage so the system gets more decisions right over time.
Design validation, retries, fallbacks, uncertainty handling, and human review paths for ambiguous, adversarial, incomplete, or conflicting inputs.
Build datasets, regression tests, dashboards, traces, and review loops that make model quality visible and prevent repeated failures.
Help debug live issues, investigate regressions, improve runbooks, and reduce repeated operator friction.

Requirements

What you’ll need

3+ years of professional software engineering experience in Python, TypeScript, or similar production languages.
Hands-on experience building production systems that use LLMs, agents, retrieval, structured outputs, or model-powered workflows.
Experience designing evaluations, test datasets, regression checks, quality metrics, or manual review loops for AI systems.
Strong debugging ability across APIs, databases, queues, logs, model outputs, and external data sources.
Practical understanding of prompt engineering, tool calling, structured output validation, retrieval, and common LLM failure modes.
Ability to reason carefully about correctness in uncertain or adversarial environments.
High agency, strong ownership, and clear written communication.

Benefits

Comp & perks

Pay packages include competitive salaries & meaningful long term equity participation
Salaries for this role range from $100-200k (USD)
Will pay in stablecoins or fiat
Philosophies for a culture that show we care: Take vacation when you need it, family care, training and development (just to name a few)
100% remote, which means we encourage you to create the work environment that you thrive in
At least two team wide offsites a year