ScienceLogic

Data Scientist

• Design and own evaluation harnesses for LLM and agentic outputs, including golden sets, regression suites, and rubric-based scoring
• Build and calibrate LLM-as-judge pipelines and validate judges against human labels
• Define and track response-quality metrics including groundedness, hallucination rate, relevance, completeness, instruction-following, and persona adherence
• Curate, version, and expand evaluation datasets
• Benchmark local models and quantify quality tradeoffs versus larger alternatives
• Red-team systems for prompt injection, jailbreaks, tool misuse, and edge cases
• Design chaos and stress tests for model and agent reliability
• Characterize failure modes and improve guardrails and regression coverage
• Evaluate retrieval quality and experiment with chunking, indexing, and hybrid retrieval
• Analyze multi-step agent trajectories, tool-call correctness, efficiency, replayable state, and guardrail breaches
• Assess intent classification and routing quality
• Monitor behavioral regressions and production output-quality drift
• Recommend fixes through prompts, retrieval, grounding, routing, and model selection
• Build, deploy, monitor, recalibrate, and retrain production models for capacity forecasting, anomaly prediction, and early-warning signals
• Define operational accuracy and lead-time metrics
• Integrate predictive signals into LLM and agentic layers for reasoning, advisories, and recommendations
• Apply AIOps/NOC analysis to log anomalies, event correlation, root-cause analysis, and problem analysis
• Quantify system economics, token consumption, interaction types, MTTR, and operator-hours
• Communicate analysis to engineering and product stakeholders
• Use LLM-assisted workflows, synthetic evaluation cases, and bootstrapped labeled data
• Track and adopt evaluation, retrieval, and agentic-analysis techniques