You will be the single owner of our product platform and release-quality systems.
Product analytics & instrumentation: Define and maintain a consistent event schema, Implement analytics across frontend, backend, and AI agents, Build internal dashboards for funnels, usage, drop-offs, and failures.
Internal & admin tooling: Build dashboards and tools for debugging sessions and inspecting AI behaviour, Replace ad-hoc scripts and database queries with reliable internal systems.
Product quality & release systems: Own feature flags, rollouts, and release checks, Define what is “shippable” for non-trivial changes, Act as a release gate owner, with authority to block unsafe releases.
AI quality & evaluation: Version and manage prompts and AI configuration changes, Define when AI evals are required before shipping, Build and maintain AI evaluation pipelines and make results visible over time.
Platform health & evolution: Define system-level health and reliability metrics, Establish expected variance vs unacceptable deviation for AI systems, Reduce platform-level debt that slows shipping and decision-making.

Requirements

At least 4 years of experience in platform, quality, or reliability-focused roles
Strong system-level thinking (not just test scripts)
Experience testing AI or agent-based systems
Understands why deterministic testing is hard — and how to work around it
Comfortable turning vague problems into clear, shipped systems
Willing to make and defend platform-level decisions
Strong product judgement and pragmatism
Experience designing internal platforms used by other engineers is nice to have
Exposure to ISO / SOC / enterprise readiness environments is nice to have
Strong full-stack fundamentals
Testing & CI: pytest; vitest; Playwright (or equivalent) for E2E and behavioural testing, GitHub CI/CD gate design
Data & Systems: Distributed and asynchronous systems, PostgreSQL
Cloud & Security: AWS logging and metrics (e.g. CloudWatch), Baseline application security testing, Familiarity with common API, auth, and WebSocket risks, OWASP ZAP or equivalent tooling
AI / Agent systems experience: Experience testing AI-, agent-, or other non-deterministic systems
Familiarity with behavioural regression detection (not brittle assertions)
Understanding why deterministic testing breaks down — and how to design around it
Experience with LLM or agent observability tools (e.g. LangSmith) is a strong plus
Experience with vision or multimodal systems is a strong plus
Experience with real-time systems (voice, streaming, WebSocket) is a strong plus.

Benefits

competitive salary
high autonomy
relocation package

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

platform ownershipproduct analyticsevent schemaAI evaluation pipelinesdeterministic testingfull-stack fundamentalspytestPostgreSQLAWSreal-time systems

Soft Skills

system-level thinkingproduct judgementpragmatismdecision-makingproblem-solvingdefending platform-level decisionsclear communication

Certifications

ISOSOC