Insights and Measurement Research Analyst
SuperAwesome
PythonSQLTypeScript
Job Level
JuniorMid-Level
Tech Stack
PythonSQLTypeScript
About the role
- Own the text-to-SQL accuracy problem end-to-end: design evals, iterate prompts, and improve retrieval/routing
- Build and operate the experimentation and evaluation loop (automatic evals, regression suites, dataset curation)
- Design pragmatic LLM application architectures (RAG, agent routing, tool-use orchestration) optimized for accuracy and latency
- Ship production-grade code and support deployments; instrument, monitor, and troubleshoot model behavior in real customer environments
- Partner closely with engineering and customers to improve semantic models, SQL generation, and data alignment
- Create feedback loops from users to systematically capture issues and convert them into measurable improvements
- Contribute to automation of environment provisioning and dev workflows to enable fast iteration
- Deliver measurable improvements in text-to-SQL accuracy, reliable CI eval/regression pipelines, clear architecture, and fast partner feedback cycles
Requirements
- 2+ years in ML/AI or data-focused engineering or data science roles building production systems data or AI systems
- Demonstrated experience tuning LLM applications: prompt engineering, evals, retrieval, agent design, or similar
- Strong hands-on coding in Python or TypeScript (TypeScript familiarity a plus; willingness to work across the stack required)
- ML engineering mindset beyond notebooks: testing, CI, observability, performance, and deployment in production
- Comfort with SQL and complex data modeling; familiarity with data warehouses and pipelines
- Pragmatic, product-oriented approach—optimize for impact over novelty; complement existing systems rather than rebuild from scratch
- Ability to design experiments, quantify improvements, and communicate trade-offs clearly
- Experience with text-to-SQL systems, semantic layers, or BI/analytics workflows (nice to have)
- Exposure to RAG frameworks, knowledge graphs, vector stores, and evaluation tooling (nice to have)
- Prior work in analytics engineering or data engineering environments (nice to have)
