LLM Engineer

• Design and build GenAI-powered developer tooling: prompts, IDE integrations, Cursor/Claude rules, and telemetry to understand how they’re used.
• Build and run the GenAI developer platform: Tools, services, and integrations leveraging LiteLLM to power both local and background/long-running agents, providing auth, configuration, and LLM routing.
• Own guardrails and observability for agentic workflows: integrate with Datadog, DX, and FinOut, define sensible standards, and make sure agents interact with Jira, GitHub, and internal platforms in a safe, auditable way.
• Work closely with engineers and managers across Product Engineering to understand how they build software today and to design solutions that actually fit the way they work.
• Support our mission and values, including journalistic independence, and keep ethics and safety central to how we use AI in developer workflows.
• This role includes limited on-call responsibilities; the schedule will be set when you join. Demonstrate support and understanding of our value of journalistic independence and a strong commitment to our mission to seek the truth and help people understand the world.
• You will report to the Engineering Manager of the DevEx Platforms team

Senior Engineer, Developer Platforms, Generative AI

• Partner with a cross-functional team of engineers, research scientists, technical program managers, and product managers to deliver AI-powered products that change how our associates work and how our customers interact with Capital One.
• Design, develop, test, deploy, and support AI software components including foundation model training, large language model inference, similarity search, guardrails, model evaluation, experimentation, governance, and observability, etc.
• Leverage a broad stack of Open Source and SaaS AI technologies such as AWS Ultraclusters, Huggingface, VectorDBs, Nemo Guardrails, PyTorch, and more.
• Invent and introduce state-of-the-art LLM optimization techniques to improve the performance — scalability, cost, latency, throughput — of large scale production AI systems.
• Contribute to the technical vision and the long term roadmap of foundational AI systems at Capital One.

Senior Lead AI Engineer, FM Hosting, LLM Inference

• Build and maintain agentic systems that continuously collect, process, and surface signals from social platforms, on-chain data, and ads ecosystems. Turn raw, multi-source data into structured intelligence that informs real account and campaign decisions.
• Design and maintain the pipelines that power those systems: ingesting from multiple external APIs, enriching and modeling data, and building semantic layers that both humans and AI agents can actually use.
• Build multi-step AI agents that automate research, synthesis, monitoring, and reporting. Not single-call LLM wrappers, but orchestrated systems with memory, defined tools, and structured output schemas that teams can depend on.
• Build full-stack applications and dashboards that make intelligence accessible across client accounts. Live tools, not decks.
• Spend real time with internal teams across marketing, PR, content, growth, and social before writing a line of code. Map how work actually moves, identify where signal gaps create friction, and build systems that solve for multiple teams from a single architecture.
• Stay ahead of the AI tooling landscape. Know when a no-code connector is the right call and when you need to build. Define how impact gets measured, and build the reporting to prove it.

AI Infra Engineer

• Design, build, and maintain scalable ML infrastructure and pipelines supporting model training, deployment, monitoring, governance, and lifecycle management.
• Develop and optimize CI/CD pipelines for machine learning and AI workloads across development, staging, and production environments.
• Build reusable ML platform capabilities including feature stores, model registries, experimentation frameworks, artifact management, and deployment automation.
• Implement scalable orchestration and workflow solutions for batch and real-time ML inference workloads.
• Create robust monitoring systems to measure model performance, detect model drift, monitor data quality, and ensure production reliability.
• Develop automation tools and self-service capabilities to improve the efficiency, scalability, and reliability of MLOps processes.
• Collaborate with Data Scientists and Software Engineers to streamline the ML lifecycle from experimentation through enterprise production deployment.
• Apply software engineering best practices to AI/ML systems including testing, observability, resiliency, security, versioning, and infrastructure-as-code.
• Identify gaps and improvement opportunities within the organization’s ML platform ecosystem and architect scalable solutions to address them.
• Support enterprise AI governance, compliance, auditability, and model risk management requirements.
• Ensure platform scalability, reliability, security, and operational excellence across AI/ML systems.
• Lead the architecture, design, and deployment of enterprise Generative AI solutions leveraging LLMs, foundation models, and agentic AI systems.
• Design and implement Retrieval-Augmented Generation (RAG) pipelines using vector databases, embeddings, semantic search, reranking, and retrieval optimization strategies.
• Build scalable LLM orchestration frameworks using technologies such as LangChain, LlamaIndex, Semantic Kernel, or equivalent frameworks.
• Develop advanced prompt engineering strategies, prompt chaining, context management, and agent workflows to improve LLM accuracy and reliability.
• Evaluate and implement fine-tuning, parameter-efficient tuning, and prompt-based optimization approaches for domain-specific use cases.
• Build AI evaluation and benchmarking frameworks to measure hallucination rates, response quality, grounding accuracy, toxicity, bias, latency, and business performance metrics.
• Implement AI safety guardrails, governance controls, content filtering, and responsible AI practices for enterprise healthcare environments.
• Design scalable GenAI APIs and microservices supporting high-throughput enterprise AI applications.
• Optimize GenAI systems for cost, latency, throughput, and inference performance across cloud and hybrid environments.
• Integrate enterprise data sources, healthcare systems, and knowledge repositories into secure GenAI workflows.
• Research and evaluate emerging GenAI technologies, open-source frameworks, and foundation models to drive innovation and continuous improvement.
• Develop architecture diagrams, technical roadmaps, implementation strategies, and executive-level documentation for enterprise AI initiatives.
• Collaborate with cybersecurity, compliance, and infrastructure teams to ensure secure and compliant deployment of GenAI solutions involving PHI and sensitive healthcare data.
• Contribute to the development of AI platform standards, reusable GenAI accelerators, templates, and engineering best practices.

Senior MLOps, Generative AI Engineer

• Define the resilience platform, own the product roadmap and delivery for specific platform features
• Collaborate with the open-source developer community — prioritizing GitHub issues, gathering feedback, supporting contributors, and channeling community signal back into the roadmap
• Collaborate with engineering on feature design, prioritization, execution, and architecture tradeoffs
• Align cross-functionally with other Product, Engineering, Product Marketing, and Field teams on requirements, roadmaps, messaging, and engagements

Technical Product Manager, AI Infrastructure Resilience

• Design end-to-end AI solutions on Dataiku's platform, leveraging Dataiku Agent Hub, Prompt Studio, LLM Mesh, and Knowledge Banks (Vector Stores), or Python-based frameworks where needed.
• Build and orchestrate multi-agent systems using Dataiku's Visual Agents (simple and structured), as well as code-based frameworks (LangGraph, CrewAI, Claude Agent SDK, OpenAI Agents SDK) as appropriate.
• Integrate and optimize LLM APIs across providers (OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure, open-source models via Dataiku's LLM Mesh), applying model routing strategies to balance cost, latency, and quality.
• Implement Retrieval-Augmented Generation (RAG) pipelines, including agentic RAG and GraphRAG, using Dataiku's Knowledge Banks with reranking, dynamic filtering, and document extraction capabilities.
• Work exclusively with the Marketing organisation, partnering across functions such as Demand Generation, Content Marketing, Product Marketing, Field Marketing, Marketing Operations, Brand, and Communications.
• Engage marketing stakeholders to gather business requirements, then go further: identify the underlying user or team pain points those requirements represent, and design solutions that address both the stated need and the deeper problem.
• Own projects end-to-end, from requirements intake and solution design through to build, deployment, and handover.
• Develop autonomous and semi-autonomous AI agents, using Dataiku's Agent Builder, custom Python-based architectures (LangGraph, CrewAI, Claude Agent SDK, etc.), or a combination of both. Exercise judgment on when to leverage platform capabilities and when to build custom solutions.
• Design and build Agent Tools beyond documented examples, including custom API integrations, data retrieval modules, decisioning logic, and automated workflows, pushing past out-of-the-box patterns to deliver solutions tailored to specific business problems.
• Build, publish, and consume MCP (Model Context Protocol) servers to enable agent-to-tool integration across systems, including designing custom MCP servers where needed.
• Develop evaluation and monitoring approaches for agent systems, combining Dataiku's built-in capabilities with custom instrumentation to measure reliability, accuracy, cost, and business impact in production.
• Design and maintain evaluation frameworks (evals) for LLM-based systems, measuring accuracy, latency, cost, and reliability in production.
• Adhere to data governance, security, and regulatory compliance requirements (EU AI Act awareness, responsible AI practices) for all AI solutions.
• Leverage Dataiku's Cost Guard and Quality Guard features to manage LLM spend, enforce usage policies, and maintain output quality standards.
• Work closely with analytics and data engineering teams to maintain metadata on reference datasets for LLM consumption.
• Create front-end user interfaces for AI applications using HTML, CSS, and JavaScript, within Dataiku's webapps framework, Dataiku Answers for chat-based interfaces, or standalone applications built with Vue.js and Node.js.
• Collaborate on UX design, ensuring internal stakeholders find AI solutions intuitive and responsive.
• Provide product feedback to the development team to improve the platform.
• Stay current with the rapidly evolving AI engineering landscape, agent frameworks, model capabilities, evaluation practices, governance requirements, and tools like MCP and A2A protocols.

LLM Engineer jobs in New York

See all jobs on JobTailor

Senior Engineer, Developer Platforms, Generative AI

Senior Lead AI Engineer, FM Hosting, LLM Inference

AI Infra Engineer

Senior MLOps, Generative AI Engineer

Technical Product Manager, AI Infrastructure Resilience

Generative AI Engineer