USAN, Inc.

GenAI Tuning Analyst

USAN, Inc.

full-time

Posted on:

Location Type: Hybrid

Location: Norcross • United States

Visit company website

Explore more

✨ AI Apply
Apply

About the role

  • Lead Reinforcement Learning from Human Feedback (RLHF) cycles.
  • Review AI-generated outputs and 'grade' them based on accuracy, empathy, and adherence to compliance protocols.
  • Define and track AI-specific KPIs, such as Hallucination Rate, Refusal Rate, and Intent Accuracy, alongside traditional metrics like First Contact Resolution (FCR).
  • Implement and monitor safety filters to ensure the AI remains neutral, avoids prohibited topics, and protects PII (Personally Identifiable Information).
  • Partner with Subject Matter Experts (SMEs) in Legal, Compliance, and Operations to translate business policy into technical model constraints.
  • Iterate on complex system prompts to guide LLM behavior for specific contact center use cases (e.g., automated summaries, live chat responses, and knowledge base retrieval).
  • Identify and curate high-quality 'golden datasets' from historical call transcripts and chat logs to be used for fine-tuning and few-shot prompting.

Requirements

  • 2+ years in AI/NLP, Data Analysis, high-level Contact Center Operations.
  • Analyzing Intent Accuracy to identify gaps in the AI’s natural language understanding (NLU/NLP).
  • UAT Support: Experience assisting in User Acceptance Testing for new AI features or tool rollouts.
  • Secondary experience in Quality Assurance is a plus.
  • Proficiency in prompt engineering techniques (Chain-of-Thought, Few-Shot, Zero Shot).
  • Basic understanding of LLM architectures (GPT-4, Claude, Llama).
  • Exceptional written and verbal communication skills with an obsession over tone, grammar, and brand consistency.
  • Ability to act as a bridge between technical teams and frontline operations.
  • Moderate customer meetings to align with company direction on our AI tools.
Benefits
  • Healthcare benefits
  • 401K plan
  • Paid company holidays
  • Paid vacation
  • Business casual work environment
  • Annual performance based bonus program
Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools
Reinforcement Learning from Human FeedbackAI-generated output evaluationKPI trackingsafety filters implementationprompt engineeringnatural language understandingUser Acceptance TestingQuality AssuranceLLM architecturesdata analysis
Soft Skills
written communicationverbal communicationattention to detailcollaborationproblem-solvingcustomer engagementinterpersonal skillsorganizational skillsadaptabilitybridge between technical and operations