Palm Ventures

Principal Engineer

Palm Ventures

full-time

Posted on:

Location Type: Hybrid

Location: Redwood City • California • United States

Visit company website

Explore more

✨ AI Apply
Apply

Job Level

Lead

Tech Stack

GraphQLKubernetesPythonSDLCSpark

About the role

  • Design and implement multi-agent systems and orchestration layers.
  • Build and operate observability stacks (e.g., OpenTelemetry) to monitor agent reasoning paths, tool usage, and performance in real-time.
  • Develop and enforce technical safety mechanisms—such as input/output filtering and behavioral boundaries—to mitigate risks like hallucinations, prompt injections, and bias.
  • Analyze telemetry and execution traces to create feedback loops for continuous agent improvement and automated evaluation.
  • Securely connect agents to external services, unstructured data, and enterprise APIs via robust tool-calling schemas.
  • Implement fallback mechanisms, human-in-the-loop (HITL) checkpoints, and automated recovery for agentic failures.
  • Implement best practices for MLOps, monitoring, and performance tuning of AI models in live environments
  • Automate SDLC processes and CI/CD pipelines, elevate QA standards, and develop incident response protocols to enable high velocity, availability and reliability of our platform

Requirements

  • 10+ years of senior engineering experience at a fast-paced, high-growth technology startup that has successfully scaled from early stage through Series A/B funding (or equivalent growth phase)
  • 5+ years of ML, including 2+ years focused on LLMs or agentic workflows.
  • Proficiency in agent orchestration and memory-augmented systems.
  • Hands-on experience analyzing tracing and logging data.
  • Experience using feedback loops to continuously improve ML systems
  • Built agents that invoked tools or utilized Model Context Protocol (MCP) to access enterprise data sources
  • Proficiency in modern technologies (e.g., Python, semantic search, vector DBs, GraphQL, queues, containers, Kubernetes, real-time data processing, Spark, Open Telemetry, Clickhouse)
  • Thrives in startup ambiguity while maintaining the discipline of an enterprise-grade engineer
  • Acts as a force multiplier who elevates the technical bar for the entire team
  • Obsessed with practical application of AI systems and capable of building enterprise solutions that solve real-world customer problems
Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools
multi-agent systemsorchestration layersobservability stacksinput/output filteringbehavioral boundariesMLOpsCI/CD pipelinesreal-time data processingsemantic searchvector DBs
Soft Skills
leadershipproblem-solvingadaptabilityteam collaborationdisciplineinnovationcommunicationanalytical thinkingtechnical mentorshipcustomer focus