Senior Software Engineer, Inference Platform

aion

Senior Software Engineer designing and building inference platforms for AI workloads. Contributing to core services and engineering excellence at a fast-growing startup.

Posted 7/14/2026full-timeLondon • 🇬🇧 United KingdomSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and building scalable AI inference service platforms, with a strong focus on distributed systems, low-latency performance, and robust deployment strategies. Proficient in implementing observability and monitoring solutions to ensure high-performance AI model serving.

Highest-signal resume keywords

AI/ML Inference SystemsDistributed Systems DesignGolang ProgrammingContainer OrchestrationObservability Tools

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

AI GatewayResource OrchestratorRuntime EnginesAutoscalerLow-Level DesignsDeployment PipelinesGPU UtilizationTelemetry StackCode ReviewsModel Lifecycle Management

Tools & Technologies

KubernetesDockerKafkaRabbitMQPostgreSQLRedisPrometheusGrafanaOpenTelemetryTensorRT-LLM

Industry Keywords

Microservices ArchitectureAPI Gateway PatternsAutoscaling StrategiesLoad BalancingResource Scheduling Algorithms

Tech Stack

Tools & technologies

Distributed SystemsDockerGoGrafanaKafkaKubernetesMicroservicesPostgresPrometheusPythonRabbitMQRedisRust

About the role

Key responsibilities & impact

Design and build aion's inference service platform the backbone for serving AI models at scale across diverse workloads
Own and architect core platform components: AI Gateway, Resource Orchestrator, Runtime Engines, and Autoscaler
Design highly modular, scalable, and extensible low-level designs (LLDs) for inference infrastructure components
Lead high-level design discussions, establish architectural patterns, and drive technical decision-making for the inference stack
Understand and optimize the dynamics of model deployment, version upgrades, and rollback strategies
Build robust deployment pipelines for seamless model updates with zero-downtime deployments
Design intelligent routing systems for multi-model serving, A/B testing, and canary deployments
Implement strategies for efficient GPU utilization and model cold-start optimization
Implement highly performant and optimized software for low-latency, high-throughput inference serving
Build and debug production-grade code in distributed systems handling real-time AI workloads
Optimize inference pipelines for latency, throughput, batching efficiency, and resource utilization
Design fault-tolerant systems with graceful degradation and automatic recovery mechanisms
Build high-performance telemetry and observability stack for inference metrics, performance tracking, and debugging
Implement comprehensive monitoring for model latency, throughput, error rates, GPU utilization, and cost per inference
Conduct thorough code reviews to maintain code quality, performance standards, and architectural consistency
Establish engineering best practices for testing, documentation, and production readiness.

Requirements

What you’ll need

4+ years of experience building and scaling backend systems, distributed platforms, or inference infrastructure
Strong understanding of AI/ML inference systems and experience with inference engines (vLLM, TGI, TensorRT-LLM, or similar)
Deep knowledge of distributed systems design, microservices architecture, and API gateway patterns
Proficiency in Golang strongly preferred; Python, Rust, C++ for performance-critical components a plus
Experience with container orchestration (Kubernetes, Docker) and infrastructure-as-code
Solid understanding of autoscaling strategies, load balancing, and resource scheduling algorithms
Experience building high-throughput, low-latency systems with sub-100ms response time requirements
Familiarity with message queues (Kafka, RabbitMQ), databases (PostgreSQL, Redis), and event-driven architectures
Knowledge of GPU computing, model serving optimizations (batching, quantization, multi-tenancy), and resource allocation
Experience with observability tools (Prometheus, Grafana, OpenTelemetry) and distributed tracing
Understanding of API design, rate limiting, authentication/authorization, and security best practices
Exposure to AI model deployment workflows and model lifecycle management is highly desirable

Benefits

Comp & perks

High ownership, self driven and biased for action.
Strong strategic thinking and ability to connect technical decisions to business impact.
Excellent communication and mentoring skills.
Thrives in ambiguity, fast-paced environments, and early-stage startup culture.
Work directly with high-pedigree founders shaping technical and product strategy.
Build infrastructure powering the future of AI computers globally.
Significant ownership and impact with equity reflective of your contributions.
Competitive compensation, flexible work options, and wellness benefits