Data Manager
Excelya
Job Level
JuniorMid-Level
Tech Stack
PythonSQL
About the role
- Think as an Engineer to develop systems that can scale and allow for fast, reliable, and trusted iterations.
- Apply scientific and engineering principles to prompt design, LLM evaluation processes, and technical integrations with ecommerce platforms and tools.
- Design, develop, and refine prompts to guide the AI Agent in generating accurate and contextually relevant responses to shopper inquiries.
- Identify patterns, and drive improvements in both prompt design and overall AI Agent performance.
- Analyze large-scale data from LLM evaluations to derive insights, Identify heuristics to sample improvement and non-regression dataset.
- Build reliable estimators for on-going AB tests, analyse optimal experiment stoping time and mitigate cross AB test contamination.
- Conduct extensive testing of prompts to ensure they perform well in diverse scenarios.
- Design and implement robust methodologies for assessing LLM performance at scale.
- Work product managers to understand requirements, incorporate feedback into prompt design and improvement, and integrate findings from large-scale evaluations into the AI Agent's development.
- Work with ML engineers to build and fine-tune in-house LLMs that out-perform state of the art.
- Maintain comprehensive documentation for prompt design, usage, best practices, and evaluation methodologies.
- Stay up-to-date with advancements in AI, natural language processing (NLP), and prompt engineering techniques.
- Analyze and interpret user interactions and large-scale performance data to identify areas for enhancement.
- Develop and refine metrics for evaluating prompt and LLM performance at scale.
- Develop and scale connections between AI Agents and the e-commerce ecosystem, allowing the AI to perform a wide range of actions.
Requirements
- 2 years of experience in statistics, prompt engineering and LLM pipelines.
- Proficiency in programming languages such as Python/SQL, large-scale data analysis, and prompt engineering tools and frameworks.
- Strong problem-solving abilities with a keen eye for detail and an understanding of how to work with AI to create a system with business impact.
- Excellent written and verbal communication skills, with the ability to articulate complex concepts to both technical and non-technical stakeholders.
- Masters degree in STEM (science, technology, engineering and mathematics), or a related field.
- Are you currently based in Paris ? (application form question suggests requirement)
- Are you legally authorized to work in the job location? (application form question suggests requirement)
- Can you commit to a hybrid work environment that involves working in the office 2 days per week? (Wednesday & Thursday)
