Docker, Inc

Senior Software Engineer, AI Gateway

Docker, Inc

full-time

Posted on:

Origin:  • 🇨🇦 Canada

Visit company website
✨ AI Apply
Manual Apply

Salary

💰 $189,600 - $260,700 per year

Job Level

Senior

Tech Stack

AWSCloudDockerGoKubernetesMicroservicesRust

About the role

  • Design and implement core systems powering the AI Gateway, including the model router, MCP gateway, and control plane
  • Build infrastructure that supports dynamic model selection, auto-failover, cost-based routing, and policy enforcement
  • Own critical capabilities such as secure credential storage, session summarization, caching, and rate limiting
  • Develop APIs for developers building with OpenAI-compatible interfaces and the Model Context Protocol
  • Build the underlying infrastructure to support evaluation, telemetry, replay, and backtesting for agents and LLM workflows
  • Lead architectural decisions and mentor engineers as the team scales
  • Collaborate with product and design to create delightful experiences in the control plane UI
  • Contribute to roadmap planning, technical strategy, and cross-functional alignment
  • Create a unified abstraction layer across diverse model and tool providers and implement secure, scalable identity and credential vaulting
  • Create infrastructure for real-time and historical analytics and ensure policy enforcement and logging end-to-end

Requirements

  • 6+ years of backend engineering experience with production-grade systems
  • Deep knowledge of distributed and highly scalable systems, cloud-native infrastructure, and API design
  • Experience building secure, high-throughput services (e.g., gateways, proxies, load balancers, policy engines)
  • Fluency in Go, and/or Rust (both preferred)
  • Familiarity with AI/ML platforms or model serving infrastructure
  • Strong product mindset and developer-facing tools experience
  • Ownership mentality with a bias for shipping, learning, and iterating
  • Prior experience with OpenAI, Anthropic, or similar LLM APIs (bonus)
  • Familiarity with RAG architectures, vector databases, or agent frameworks (e.g., LangChain, AutoGPT, CrewAI) (bonus)
  • Experience with policy engines (e.g., OPA), observability frameworks (e.g., OpenTelemetry), or API gateways (bonus)
  • Understanding of OAuth2.1, secret management, and cloud IAM systems (bonus)
  • Experience with Kubernetes, Docker, and microservices architecture (bonus)
  • Due to the remote nature of this role, unable to provide visa sponsorship