Senior Backend Engineer

Instrumentl

Senior Backend Engineer developing AI tools for nonprofits at Instrumentl. Building production backends and collaborating with product teams for grant discovery and application drafting.

Posted 6/26/2026full-timeRemote • 🇺🇸 United StatesSenior💰 $175,000 - $220,000 per yearWebsite

Tech Stack

Tools & technologies

AWSDockerGoogle Cloud PlatformNode.jsPythonRubyRuby on RailsSQLTypeScript

About the role

Key responsibilities & impact

Ship AI to production
Build tool-using LLM agents (task planning, function and tool calling, multi-step workflows, guardrails) for grant discovery, application drafting, and research assistance.
Turn prototypes into resilient, observable services with clear SLAs, rollback and fallback strategies, and cost and latency budgets.
Stand up evaluation and observability so our AI stays grounded, safe, and cost-effective.
Build trustworthy backends
Write high-quality, thoroughly tested code across the backend and the data pipelines that power retrieval and evaluation.
Contribute to reliability practices: alerts, dashboards, and incident response.
Collaborate and raise the bar
Partner with Product, Design, and GTM on scoping, UX, and measurement.
Run experiments (A/B, canaries), interpret results, and iterate.
Raise engineering standards through clear, maintainable code, tests, docs, and thoughtful review.

Requirements

What you’ll need

7+ years building and shipping production backend systems in Python (FastAPI, Celery, or equivalent), taking features from prototype to production with real reliability practices like tests, observability, and rollback.
Hands-on experience building LLM features in production: tool and function calling, multi-step agent workflows, and the guardrails and evals that keep them grounded, safe, and cost-effective. This is the core of the role.
Strong data fundamentals: SQL, schema design, and building pipelines that power retrieval and evaluation.
Thrives in a fast, scrappy startup environment with high ownership and a bias for action, speed, quality, and simplicity.
Nice to have
TypeScript and Node, plus familiarity with Ruby on Rails (our core platform) or a willingness to learn it.
Experience with AWS or GCP, Docker, CI/CD, and observability (logs, metrics, traces).
RAG depth: document ingestion, chunking and windowing, embeddings, hybrid search (keyword plus vector), re-ranking, and grounded citations.
Experience with re-rankers and cross-encoders, hybrid retrieval tuning, or search and recommendation systems.
Evaluation mindset: designing eval suites (RAG/QA, extraction, summarization) using automated and human-in-the-loop methods, with familiarity with frameworks like Ragas, DeepEval, or OpenAI Evals.
Orchestration frameworks: LangChain or LangGraph, LlamaIndex, Semantic Kernel, or custom orchestration.

Benefits

Comp & perks

100% covered health, dental, and vision insurance for employees (50% for dependents)
Generous PTO, including parental leave
401(k)
Company laptop and home-office stipend
Bi-annual company retreats
Instrumentl is evolving rapidly. You’ll always have new challenges and opportunities to grow here.

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

PythonFastAPICelerySQLdata pipelinesTypeScriptNodeRuby on RailsDockerCI/CD

Soft Skills

ownershipbias for actionspeedqualitysimplicity