Lead AI Engineer
Vytwo Technologies Inc
full-time
Posted on:
Location Type: Hybrid
Location: Bengaluru • India
Visit company websiteExplore more
Job Level
About the role
- Design and build end-to-end data and AI pipelines using Azure Databricks.
- Develop robust ETL/ELT workflows using Python (PySpark) and SQL.
- Implement CI/CD pipelines for Databricks deployments (jobs, notebooks, workflows).
- Integrate Databricks with Azure services (Data Lake, Blob Storage, Key Vault, Azure OpenAI, Azure Functions, etc.).
- Optimize jobs for performance, cost, and reliability.
- Build reusable, modular code.
- Collaborate with data scientists and platform teams to move models from experimentation to production.
- Implement logging, monitoring, and error handling for production pipelines.
- Develop and deploy ML and Generative AI models (LLMs, embeddings, RAG pipelines) for NLP, computer vision, and predictive analytics.
- Fine-tune LLMs using LoRA/QLoRA and integrate with Azure OpenAI or Hugging Face models.
- Implement vector search and retrieval pipelines using FAISS or Azure Cognitive Search.
- Ensure responsible AI practices, including bias detection and model governance.
Requirements
- Azure Databricks (jobs, workflows, clusters, Unity Catalog preferred).
- Python (PySpark-heavy, not just pandas).
- SQL (complex joins, window functions, analytical queries).
- Azure Cloud (ADLS Gen2, ADF, Key Vault, IAM concepts).
- Pipeline orchestration & deployment (CI/CD, environment promotion).
- Azure DevOps.
- Strong understanding of ML lifecycle and MLOps best practices.
- Experience with model deployment using MLflow or similar frameworks.
Benefits
- Flexible work from home options
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills & Tools
PythonPySparkSQLETLELTCI/CDMLGenerative AINLPcomputer vision
Soft Skills
collaborationproblem-solvingcommunicationorganizational skillsattention to detail