Prompt Engineer, LLM Systems, Evals, Safety
Supertech Innovation Labs
full-time
Posted on:
Location Type: Remote
Location: Remote • 🇮🇳 India
Visit company websiteJob Level
Mid-LevelSenior
Tech Stack
PythonTypeScript
About the role
- Design high-quality prompts, system instructions, and tooling that make our LLM features accurate, safe, and cost-effective.
- Own evaluation, prompt versioning, and continuous improvement.
- Author, refactor, and chain prompts (system/tool/policy) for varied tasks.
- Create offline/online evaluation harnesses (rubrics, golden sets, metrics).
- Build prompt libraries with versioning, A/B testing, and telemetry.
- Reduce hallucinations via verification, constrained decoding, and tool use.
- Implement safety: jailbreak/prompt-injection tests, content policy checks, PII handling.
- Partner with engineers to integrate prompts into production features.
Requirements
- Demonstrated prompt design across multiple task types and models.
- Experience building eval datasets and automated scoring (e.g., accuracy, faithfulness, utility, cost/latency).
- Familiarity with retrieval-augmented generation concepts and tool/function calling.
- Strong scripting (Python/TypeScript) for data prep, evals, and analysis.
- Clear writing; ability to translate business goals into measurable prompt specs.
- Nice-to-Haves
- Experience with LangChain/LLM orchestration, vector stores, and rerankers.
- Knowledge of safety tooling and red-teaming techniques.
- Experiment platforms (feature flags, A/B tests), analytics.
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard skills
prompt designevaluation datasetsautomated scoringscriptingPythonTypeScriptLangChainA/B testingdata preparationanalysis
Soft skills
clear writingtranslating business goalscommunication