LLM Engineer

• Collaborate closely with our AI and ML research teams to understand their infrastructure needs and obstacles
• Monitor and optimize the performance of our infrastructure ensuring high availability, scalability, and efficient resource utilization
• Help define and improve important measures of AI researcher efficiency, ensuring that our actions are in line with measurable results
• Collaborate with diverse teams, including researchers, data engineers, and DevOps professionals, to build a seamless and coordinated AI/ML infrastructure ecosystem
• Stay on top of the latest advancements in AI/ML technologies, frameworks, and effective strategies, and promote their implementation within the company

AI and ML Infra Software Engineer, GPU Clusters

• Build and integrate crucial public good infrastructure to support evidence-based AI-native data system implementation that strengthens data feedback loops.
• Lead a research and development portfolio to test, co-develop, and refine integrated data and AI infrastructure solutions in real-world education and workforce contexts.
• Design and lead iterative learning cycles at the intersection of the AI Infrastructure and System Feedback Loops (SFL) portfolios.
• Generate actionable evidence about what conditions, infrastructure configurations, and integrated approaches accelerate impact for learners.
• Apply mastery-level expertise in implementation research and education data systems, and a proactive, solutions-oriented perspective to navigate ambiguity and drive thorough, equity-centered inquiry in partnership with grantees, researchers, and cross-divisional colleagues.

Senior Program Officer, Data & AI Infrastructure Integration

• Design, build, and maintain scalable ML infrastructure and pipelines supporting model training, deployment, monitoring, governance, and lifecycle management.
• Develop and optimize CI/CD pipelines for machine learning and AI workloads across development, staging, and production environments.
• Build reusable ML platform capabilities including feature stores, model registries, experimentation frameworks, artifact management, and deployment automation.
• Implement scalable orchestration and workflow solutions for batch and real-time ML inference workloads.
• Create robust monitoring systems to measure model performance, detect model drift, monitor data quality, and ensure production reliability.
• Develop automation tools and self-service capabilities to improve the efficiency, scalability, and reliability of MLOps processes.
• Collaborate with Data Scientists and Software Engineers to streamline the ML lifecycle from experimentation through enterprise production deployment.
• Apply software engineering best practices to AI/ML systems including testing, observability, resiliency, security, versioning, and infrastructure-as-code.
• Identify gaps and improvement opportunities within the organization’s ML platform ecosystem and architect scalable solutions to address them.
• Support enterprise AI governance, compliance, auditability, and model risk management requirements.
• Ensure platform scalability, reliability, security, and operational excellence across AI/ML systems.
• Lead the architecture, design, and deployment of enterprise Generative AI solutions leveraging LLMs, foundation models, and agentic AI systems.
• Design and implement Retrieval-Augmented Generation (RAG) pipelines using vector databases, embeddings, semantic search, reranking, and retrieval optimization strategies.
• Build scalable LLM orchestration frameworks using technologies such as LangChain, LlamaIndex, Semantic Kernel, or equivalent frameworks.
• Develop advanced prompt engineering strategies, prompt chaining, context management, and agent workflows to improve LLM accuracy and reliability.
• Evaluate and implement fine-tuning, parameter-efficient tuning, and prompt-based optimization approaches for domain-specific use cases.
• Build AI evaluation and benchmarking frameworks to measure hallucination rates, response quality, grounding accuracy, toxicity, bias, latency, and business performance metrics.
• Implement AI safety guardrails, governance controls, content filtering, and responsible AI practices for enterprise healthcare environments.
• Design scalable GenAI APIs and microservices supporting high-throughput enterprise AI applications.
• Optimize GenAI systems for cost, latency, throughput, and inference performance across cloud and hybrid environments.
• Integrate enterprise data sources, healthcare systems, and knowledge repositories into secure GenAI workflows.
• Research and evaluate emerging GenAI technologies, open-source frameworks, and foundation models to drive innovation and continuous improvement.
• Develop architecture diagrams, technical roadmaps, implementation strategies, and executive-level documentation for enterprise AI initiatives.
• Collaborate with cybersecurity, compliance, and infrastructure teams to ensure secure and compliant deployment of GenAI solutions involving PHI and sensitive healthcare data.
• Contribute to the development of AI platform standards, reusable GenAI accelerators, templates, and engineering best practices.

Senior MLOps, Generative AI Engineer

• Define the resilience platform, own the product roadmap and delivery for specific platform features
• Collaborate with the open-source developer community — prioritizing GitHub issues, gathering feedback, supporting contributors, and channeling community signal back into the roadmap
• Collaborate with engineering on feature design, prioritization, execution, and architecture tradeoffs
• Align cross-functionally with other Product, Engineering, Product Marketing, and Field teams on requirements, roadmaps, messaging, and engagements

Technical Product Manager, AI Infrastructure Resilience

• Lead bring-up, validation, and debugging of large-scale AI clusters, infrastructure, and end-to-end workloads, setting the standard for how the team operates.
• Bring up, tune, and benchmark AI pre-training, post-training, and inference workloads using PyTorch, NeMo / Megatron, TensorRT-LLM, and adjacent NVIDIA AI software stacks.
• Profile and optimize end-to-end workload performance across compute, memory, networking, and communication layers using tools such as Nsight Systems, NCCL tests, and custom microbenchmarks.
• Analyze scaling efficiency for distributed LLM workloads using data, tensor, pipeline, and expert parallelism across modern GPU clusters, and translate findings into concrete tuning guidance.
• Own root-cause analysis of complex failures — hangs, performance regressions, topology sensitivity in large distributed environments.
• Define and build the resilience and failure-attribution stack: detecting, triaging, and attributing node, fabric, and workload failures across the cluster at scale.
• Build repeatable benchmark suites, automation, acceptance criteria, and qualification workflows on new platforms.
• Tune runtime settings, communication parameters, and deployment configurations in close partnership with framework, systems, and platform teams.
• Deliver actionable, data-driven recommendations based on profiling, benchmark results, and cluster characterization.
• Mentor engineers, drive technical standards, and act as a force multiplier across the broader performance and infrastructure organization.

Senior Software Engineer, DGX Cloud AI Infrastructure

• Bring up, validate, and debug large-scale AI clusters, infrastructure, and end-to-end workloads.
• Bring up, tune, and benchmark AI pre-training, post-training, and inference workloads using PyTorch, NeMo / Megatron, TensorRT-LLM, and adjacent NVIDIA AI software stacks.
• Perform root-cause analysis of failures in large distributed environments.
• Contribute to the resilience and failure-attribution tooling that detects, triages, and attributes node, fabric, and workload failures across the cluster.
• Build and maintain repeatable benchmark suites, automation, acceptance criteria, and qualification workflows on new platforms.
• Tune runtime settings, communication parameters, and deployment configurations in close partnership with framework, systems, and platform teams.
• Deliver actionable, data-driven recommendations based on profiling, benchmark results, and cluster characterization.

LLM Engineer jobs in Washington

See all jobs on JobTailor

AI and ML Infra Software Engineer, GPU Clusters

Senior Program Officer, Data & AI Infrastructure Integration

Senior MLOps, Generative AI Engineer

Technical Product Manager, AI Infrastructure Resilience

Senior Software Engineer, DGX Cloud AI Infrastructure

Software Engineer, DGX Cloud AI Infrastructure