STN Incorporated

Engineer

• Design, deploy, and optimize multi-node GPU clusters for AI training and inference workloads
• Tune distributed training environments to maximize GPU utilization, throughput, and scaling efficiency
• Optimize inference clusters for token generation throughput, low latency, and high GPU utilization
• Build and support production AI infrastructure running hundreds to thousands of GPUs
• Analyze and eliminate performance bottlenecks across compute, networking, storage, and software layers
• Perform NCCL benchmarking, analysis, and tuning for collective communication performance
• Design and optimize GPU networking using InfiniBand or RoCE v2, including RDMA, congestion management, topology awareness, and QoS
• Configure and tune distributed AI software stacks including PyTorch, NCCL, CUDA, UCX, MPI, Slurm, and Pyxis/Enroot
• Optimize GPU scheduling and resource allocation for training and inference environments
• Develop benchmarking and validation processes for hardware, firmware, drivers, and software releases
• Identify performance regressions and troubleshoot distributed training issues at scale
• Optimize storage architectures for checkpointing, dataset streaming, and high-performance parallel I/O
• Work closely with ML engineers to improve training scalability and inference efficiency
• Create automation to deploy, validate, benchmark, and monitor GPU clusters
• Evaluate emerging AI infrastructure technologies and recommend platform architecture improvements

Cluster Engineer