Netomi

Lead/Senior Product Manager – Analytics, Evals & Conversational BI

Netomi

full-time

Posted on:

Location Type: Remote

Location: Canada

Visit company website

Explore more

✨ AI Apply
Apply

Job Level

Senior

Tech Stack

ETL

About the role

  • Define and execute the Analytics/Evals/Governance roadmap with clear sequencing (MVP → v1 → v2) and measurable adoption targets.
  • Partner deeply with Data Science to productize evaluation methodology (what we score, how we calibrate, how we prevent “gaming,” how we track drift).
  • Partner with Engineering and Observability teams to standardize telemetry (tool spans, workflow transitions, model calls, cost/latency metrics) and make it usable in product.
  • Drive a cohesive Agentic Studio UX across Build / Operate / Improve workflows: dashboards, drill-downs, investigation flows, alerts, and remediation actions.
  • Establish objective success metrics and instrument them end-to-end (data correctness, timeliness, reliability, and customer impact).
  • Work with Delivery/CS and enterprise partners to ensure analytics is usable for real operational processes: incident response, change management, governance reviews, and quarterly business reviews.

Requirements

  • Bachelor’s degree in Computer Science, Data Science, Engineering, Statistics, or a related quantitative/technical field (Master’s a plus).
  • 6–10+ years of experience across Product Management and/or Data Science/Analytics leadership, including shipping analytics products to enterprise users.
  • Demonstrated depth in: data modeling, telemetry/event schemas, ETL/ELT concepts, and analytics correctness.
  • Building operational + business reporting (real-time monitoring and historical analytics).
  • Experimentation and measurement design (A/B testing concepts, uplift measurement, cohorting).
  • Strong hands-on experience applied to AI/LLM or data science platforms, including at least several of: LLM evaluation methods (automated scoring, multi-turn evaluation, calibration).
  • Observability for probabilistic systems (quality + latency + cost).
  • Tool calling / workflow execution analytics and failure taxonomy.
  • Governance measurement (policy outcomes, escalation triggers, compliance reporting).
  • Strong product craft: UX and information architecture instincts, ability to design operator workflows, and capability packaging that scales across customers.
Benefits
  • Health insurance
  • Retirement plans
  • Paid time off
  • Flexible work arrangements
  • Professional development

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard skills
data modelingtelemetryETLELTanalytics correctnessA/B testingcohortingAILLM evaluation methodsgovernance measurement
Soft skills
product managementleadershipUX designinformation architectureworkflow designcollaborationcommunicationanalytical thinkingproblem-solvingcustomer impact focus