Infrastructure Engineer

aion

Infrastructure Engineer designing observability systems for GPU infrastructure. Joining ambitious AI startup to democratize AI compute and create elegant, accessible systems.

Posted 7/14/2026full-timeBengaluru • 🇮🇳 IndiaMid-LevelSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in GPU observability and monitoring systems, with a strong focus on deploying and managing production-scale observability tools like Loki, Grafana, and Prometheus. Proficient in systems programming and Kubernetes controller development, ensuring effective monitoring and resource management for AI workloads and HPC environments.

Highest-signal resume keywords

GPU Observability ExpertiseLoki, Grafana, Prometheus ProficiencySystems Programming in Go or RustKubernetes Controller DevelopmentHPC Systems Experience

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

GPU MonitoringPrometheus ExportersSLURM Cluster ManagementMetrics Architecture DesignInfrastructure as Code

Soft Skills

Problem-SolvingCollaborationAnalytical Thinking

Tools & Technologies

DCGMNVMLArgoCDGitOpsOpenTelemetry

Industry Keywords

AI Workload ObservabilityMulti-Tenant ObservabilityActive & Passive MonitoringNetworking ProficiencyCustom Resource Definitions

Tech Stack

Tools & technologies

GoGrafanaKubernetesPrometheusPythonRustTerraform

About the role

Key responsibilities & impact

Build and deploy comprehensive monitoring for GPU infrastructure using DCGM, NVML, and custom exporters; design metrics collection pipelines that track GPU health, utilization, thermal management, and performance across heterogeneous providers
Deploy and manage production-scale Loki, Grafana, Tempo, and Mimir alongside Prometheus and Thanos/VictoriaMetrics; design retention strategies, aggregation rules, and query patterns that scale to thousands of GPUs
Write Prometheus exporters in Go or Python for GPU metrics, platform services, and infrastructure components; implement proper metric naming, labeling strategies, and follow OpenMetrics standards
Build custom controllers and operators for GPU workload management, scheduling, and resource allocation; instrument controllers with comprehensive metrics and tracing for observability
Design and implement specialized observability for AI training workloads (GPU efficiency, distributed training performance, resource utilization) and inference services (latency percentiles, throughput, cost analytics)
Deploy and manage SLURM clusters for HPC workloads, build observability for batch jobs, create bridges between SLURM and Kubernetes, and design unified monitoring across orchestrators
Write and deploy systemd services for bare-metal GPU nodes including monitoring agents, metric collectors, and platform daemons; implement proper logging and error handling
Manage infrastructure using ArgoCD and GitOps workflows, create observable platform abstractions that teams consume declaratively, build self-service capabilities with integrated monitoring
Design intelligent, actionable alerting systems for GPU failures, thermal throttling, performance degradation, and workload anomalies; define platform SLOs and implement comprehensive monitoring to track reliability
Build secure observability isolation ensuring customers access only their metrics and logs while maintaining platform-wide visibility for operations; implement query-time filtering and RBAC
Implement automated collection of GPU and platform telemetry; integrate with OpenTelemetry for unified observability; manage cardinality and storage costs through intelligent aggregation
Build monitoring pipelines that track GPU utilization, idle time, efficiency metrics, and cost allocation per tenant; create dashboards for platform economics and provider payout calculations
Design systems that use observability data to detect hardware failures, trigger workload migration, and handle graceful degradation; ensure monitoring survives infrastructure changes
Build intuitive dashboards, APIs, and alerting interfaces enabling providers to monitor hardware contributions and customers to track workload performance in real-time
Use observability systems to rapidly identify root causes during production issues - GPU hardware failures, network bottlenecks, or workload problems; build tooling that reduces MTTD and MTTR
Leverage observability data to identify bottlenecks in distributed training, inference latency issues, networking inefficiencies, and opportunities for GPU utilization improvement

Requirements

What you’ll need

6-10 years of experience in infrastructure engineering with strong focus on observability, monitoring systems, and production service development (exceptional candidates with different experience profiles will be considered)
GPU Observability expertise with production experience monitoring NVIDIA GPUs using DCGM, NVML, and nvidia-smi; understanding GPU-specific metrics (utilization, memory, temperature, power, ECC errors) and failure modes
LGTM Stack proficiency deploying and operating Loki, Grafana, Tempo, and Mimir alongside Prometheus in production; experience with at least one long-term storage solution (Thanos, VictoriaMetrics, or Mimir)
Systems Programming in Go or Rust for building production services including custom Prometheus exporters, monitoring agents, systemd daemons, and instrumented controllers
Kubernetes controller development building custom controllers and operators using controller-runtime or client-go; implementing proper instrumentation and observability for custom resources
Active & Passive Monitoring designing SLO/SLI frameworks, implementing intelligent alerting strategies, building health check systems, and creating anomaly detection for distributed workloads
HPC Systems experience deploying and managing SLURM clusters, understanding job schedulers, and monitoring batch workloads with observability requirements different from typical web services
Advanced Kubernetes expertise including custom resource definitions, admission controllers, scheduling extensions, and cluster-wide monitoring architectures
AI Workload Observability monitoring ML training jobs (GPU efficiency, distributed training metrics, NCCL performance) and inference workloads (latency, throughput, batch processing, cost per inference)
Metrics Architecture design including cardinality management, aggregation strategies, recording rules, retention policies, and balancing observability costs with data granularity at scale
GitOps & ArgoCD experience managing observable infrastructure declaratively, building platform abstractions, and creating self-service systems with integrated monitoring
Systems expertise writing unit files, managing service dependencies, deploying monitoring agents as systemd services, and debugging service failures on bare-metal hosts
Multi-tenant observability implementing secure metrics and log isolation, RBAC policies for monitoring data, and ensuring proper isolation while maintaining platform-wide visibility
Networking proficiency with CNI plugins, understanding network observability, and monitoring distributed training communication patterns
Infrastructure as Code using Terraform or similar tools to deploy observable infrastructure, building monitoring into provisioning workflows

Benefits

Comp & perks

Founder-level ownership and bias for action.
Strong strategic thinking and ability to connect technical decisions to business impact.
Excellent communication and mentoring skills.
Thrives in ambiguity, fast-paced environments, and early-stage startup culture.
Why Join aion?
Work directly with high-pedigree founders shaping technical and product strategy.
Build infrastructure powering the future of AI compute globally.
Significant ownership and impact with equity reflective of your contributions.
Competitive compensation, flexible work options, and wellness benefits