LLMOps Platform Engineer
SumerSports
full-time
Posted on:
Location Type: Remote
Location: United States
Visit company websiteExplore more
Tech Stack
About the role
- Build and operate the LLM Platform: Develop model routing, prompt registry, and orchestration services for multi-model workflows.
- Integrate external LLM APIs (OpenAI, Anthropic, Mistral) and internal finetuned models.
- Enable fast, safe experimentation: Implement automated evaluation pipelines (offline + online) with golden sets, rubrics, and regression detection.
- Support CI/CD for prompt and model changes, with rollback and approval gates.
- Collaborate cross-functionally: Partner with product pods to instrument RAG pipelines and prompt versioning.
- Work with deep learning and data teams to integrate structured and unstructured retrieval into LLM workflows.
- Optimize performance and cost: Profile latency, token usage, and caching strategies.
- Build observability and monitoring for LLM calls, embeddings, and agent behaviors.
- Ensure reliability and safety: Implement guardrails (toxicity, PII filters, jailbreak detection). Maintain policy enforcement and audit logging for AI usage.
Requirements
- 5+ years of experience in applied ML, NLP, or ML infrastructure engineering
- Strong coding skills in Python and experience with frameworks like LangChain, LlamaIndex, or Haystack
- Solid understanding of retrieval-augmented generation (RAG), embeddings, vector databases, and evaluation methodologies
- Experience deploying models or AI systems in production environments (AWS, GCP, or Azure)
- Familiarity with prompt management, LLM observability, and CI/CD automation for AI workflows
Benefits
- Competitive Salary and Bonus Plan
- Comprehensive health insurance plan
- Retirement savings plan (401k) with company match
- Remote working environment
- A flexible, unlimited time off policy
- Generous paid holiday schedule - 13 in total including Monday after the Super Bowl
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills & Tools
PythonLangChainLlamaIndexHaystackretrieval-augmented generationembeddingsvector databasesevaluation methodologiesCI/CD automationmodel deployment
Soft Skills
collaborationcross-functional teamworkcommunicationproblem-solvingorganizational skills