AI Developer
Kaleidoscope Innovation
full-time
Posted on:
Location Type: Office
Location: Charlotte • North Carolina • United States
Visit company websiteExplore more
Tech Stack
About the role
- Build and enhance LLM/agent orchestration (Planner/supervisor patterns, tool-using agents, routing, guardrails)
- Implement intent classification information extraction validation and decision logic for servicing workflows
- Develop tool calling integrations to downstream systems (CRM, workflow engine, core banking services, case management)
- Implement human-in-the-loop workflows (review, approval, escalation, override) based on confidence/risk thresholds
- Design and implement retrieval-augmented generation (RAG) for policy procedure grounding and resolution guidance
- Build knowledge ingestion pipelines with refresh/versioning
- Improve answer quality via chunking strategies, embeddings re ranking and context management
- Define and run evaluation frameworks: golden datasets, scenario tests, regression tests, and automated scoring
- Reduce hallucinations and risk by implementing prompt policies, constraints, structured outputs, and verification steps
- Implement observability for agents (latency, cost, tool failures, drift, quality signals, escalation rates)
- Support CI/CD for agent prompts and configurations (versioning, approvals, rollback)
- Collaborate with platform and security teams on secrets management, access controls, PII protections, and safe deployments
Requirements
- 4+ years of software engineering experience or equivalent with strong CS fundamentals
- Hands-on experience building with LLMs and modern AI app stack (agents, RAG, tool/function calling)
- Strong proficiency in Python and building back-end services/APIs
- Experience with at least one: LangChain / LangGraph, Llamalndex, Semantic Kernel or equivalent frameworks
- Experience with vector databases and search (e.g., Pinecone, Weaviate, Milvus, OpenSearch/Elastic, pgvector)
- Experience deploying services in cloud environments (AWS/Azure/GCP) with basic DevOps practices
- Strong understanding of security and privacy principles (PII handling, least privilege, audit logging)
Benefits
- Health insurance
- Retirement plans
- Paid time off
- Flexible work arrangements
- Professional development
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills & Tools
LLMagent orchestrationintent classificationinformation extractiontool calling integrationsretrieval-augmented generationknowledge ingestion pipelinesevaluation frameworksPythonback-end services
Soft Skills
collaborationproblem-solvingdecision makingrisk managementcommunication