Production Engineer

• Own and evolve production operations strategy for AI/ML platforms and services
• Define SLOs, SLIs, and error budgets for AI systems
• Lead root cause analysis (RCA) and drive systemic improvements post-incident
• Establish operational readiness standards for launching new AI capabilities
• Build frameworks for on-call excellence , incident response, and escalation
• Design and implement end-to-end observability systems across AI workloads
• Implement model observability (drift detection, data skew, performance degradation)
• Build internal platforms and tooling for automated incident detection and response
• Mentor senior engineers and influence cross-team architectural decisions

Principal Software Engineer – AI Platform, Production Engineering, Reliability

• You will be part of an DGX Cloud team responsible for production systems that enable large scalable GPU clusters to be used for a variety of AI workloads.
• This includes working on custom software related to GPU asset provisioning, configuration, and lifecycle management across cloud providers.
• Implementing monitoring and health management capabilities that enable industry leading reliability, availability, and scalability of GPU assets.
• You will be harnessing multiple data streams, ranging from GPU hardware diagnostics to cluster and network telemetry.
• Working with teams across NVIDIA to ensure production AI clusters run reliability and consistently with maximum performance.
• Evaluating system failures and improving services based on a well-defined incident management process.

Senior Production Engineer – DGX Cloud

• Manage employees performing engineering and technical activities in Production Engineering
• Develop and execute integrated organizational plans, policies, and procedures
• Acquire resources for organizational activities and provide technical management
• Develop and maintain relationships with customers, stakeholders, partners, and direct reports
• Provide oversight and approval of technical approaches, products, and processes
• Work with engineering leaders to define the Production Engineering MRO Playbook

Production Engineering Leader

• Manage employees performing engineering and technical activities in the area of Production Engineering across multiple locations and business units
• Develop and execute integrated organizational plans, policies and procedures and guides the development of business and technical strategies, goals, objectives
• Acquire resources for organizational activities, provides technical management of suppliers and leads process improvements
• Develop and maintain relationships and partnerships with customers, stakeholders, peers, partners and direct reports
• Provide oversight and approval of technical approaches, products and processes
• Manage, develop and motivate employees and mid-level managers
• Work with BGS MRO Engineering Leaders, MRO Strategy Steering Team to define and Production Engineering MRO Playbook outlining job roles, MRO statement of work, skills and training, and top level MRO specific processes and tools
• Update requirements, processes and local level documentation for MRO workflow focus alignment with industry standards and requirements
• Partner to adjust management system, culture, and operating rhythm to serve MRO operating style, moving toward industry standards
• Ensure PE Design Practice Ecosystem captures all aspects of the MRO business
• Ensure PE command media taxonomy captures all aspects of the MRO business

Chief Production Engineering Manager

• Investigate and triage P1/P2 production issues with urgency and precision
• Perform deep root cause analysis across distributed systems.
• Implement immediate fixes or workarounds to restore stability.
• Clearly document findings and hand off long-term solutions to engineering teams.
• Monitor system health using tools like Grafana, logs, and telemetry dashboards.
• Identify anomalies and proactively investigate issues before escalation.
• Tune alerts and improve signal-to-noise ratio across monitoring systems.
• Partner with DevOps/SRE to strengthen reliability practices.
• Build internal tools, scripts, or small services to improve support and operations workflows.
• Contribute to codebases (primarily .NET / Azure environments) where needed.
• Help reduce technical debt by addressing recurring production issues.
• Translate technical issues into clear, actionable updates for stakeholders.
• Collaborate with Product, Engineering, and Customer-facing teams.
• Occasionally join calls with internal teams or customers to explain incidents and resolutions.

See all jobs on JobTailor

Principal Software Engineer – AI Platform, Production Engineering, Reliability

Senior Production Engineer – DGX Cloud

Production Engineering Leader

Chief Production Engineering Manager

SRE/Support Developer, Production Engineering