
Staff AI Engineer
Grafana Labs
full-time
Posted on:
Location Type: Remote
Location: United States
Visit company websiteExplore more
Salary
💰 $174,986 - $209,983 per year
Job Level
About the role
- Own end-to-end development of multi-agent AI systems, from architecture and implementation through testing, deployment, and ongoing operation
- Build modular, composable agentic systems using orchestration frameworks (LangChain, CrewAI, Anthropic MCP, or similar) that operate 24/7 across teams
- Develop reusable agentic skills that agents invoke across interfaces (Slack, dashboards, internal apps, CLIs)
- Implement observability and feedback loops including logging, performance metrics, prompt iteration, model evaluation, and cost management
- Establish governance and compliance standards for AI workflows including access controls, audit trails, PII handling, and human-in-the-loop escalation paths
- Build MCP servers, APIs, CLIs, and microservices connecting AI models to business systems (BigQuery, Slack, CRMs, email, calendars, analytics tools)
- Architect data flows for retrieval-augmented generation (RAG), connecting LLMs to internal knowledge bases, customer data, and real-time business context
- Build serverless or containerized services (GCP Cloud Functions, Cloud Run) that scale with usage and integrate with Grafana's cloud infrastructure
- Partner with RevOps, Demand Generation, Regional Marketing, and SDR teams to scope high-impact automation problems, identify bottlenecks, and build solutions with measurable business outcomes
- Design and deploy workflows using orchestration tools (n8n, Workato, or custom platforms) with CI/CD, testing, and production reliability standards
- Build systems designed for self-service with documentation, playbooks, and enablement materials that let partner teams operate independently
Requirements
- 8+ years of software engineering experience with depth in backend development, systems integration, or data/analytics engineering
- 2+ years hands-on experience applying LLMs/AI to production workflows, not just prototypes
- Strong proficiency in Python and JavaScript/Node.js with Git-based workflows, code review practices, and testing discipline
- Hands-on experience with LLM frameworks and patterns including prompt engineering, RAG, function calling/tool use, structured output parsing, and evaluation
- Experience building and operating multi-agent systems at scale including agent decomposition, orchestration patterns (sequential chains, router/dispatcher, parallel fan-out), state management, and production monitoring
- Diagnose business problems before writing code; think in workflows and outcomes, not just functions
- Deep familiarity with Google Cloud Platform, BigQuery, and serverless/containerized services (Cloud Functions, Cloud Run)
- Understanding of LLM failure modes and production mitigations including confidence thresholds, fallback logic, human escalation, and cost/latency management
- Proven ability to identify high-leverage problems, push back on low-impact requests, and deliver end-to-end with minimal direction
- Fluent with AI-assisted development tools (GitHub Copilot, Cursor, Claude Code); use AI to build AI systems
- Clear technical communicator—able to explain complex systems in simple terms to both engineers and business stakeholders.
Benefits
- equity
- bonus (if applicable)
- Restricted Stock Units (RSUs)
- 30 days annual leave
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills & Tools
PythonJavaScriptNode.jsLLMsAIbackend developmentsystems integrationdata engineeringorchestration frameworksmicroservices
Soft Skills
problem diagnosisworkflow thinkingoutcome orientationtechnical communicationindependent operationcollaborationcritical thinkingadaptabilityleadershipdocumentation