Tech Stack
AirflowCloudETLPySparkPythonSQL
About the role
- Build, scale, and maintain robust data solutions to support business objectives.
- Design, implement, and optimize high-performance data pipelines (extraction, loading, transformation, and orchestration) with scalability, reliability, and speed.
- Lead end-to-end software development projects involving large language models (LLMs), retrieval-augmented generation (RAG) frameworks, and other AI technologies.
- Champion modern software engineering practices such as CI/CD, infrastructure- as-code, containerization, and cloud-native deployments.
- Collaborate with business stakeholders to transform use cases into production-ready services and solutions, owning systems from concept to production.
- Implement rigorous testing and monitoring practices to ensure superior data quality and integrity.
- Mentor and guide junior team members, fostering a culture of excellence and continuous learning.
Requirements
- Bachelor’s degree or higher in a STEM field (Computer Science, Math, Physics, Engineering, or related), required.
- Advanced degree in Computer Science or related field, preferred.
- 5–7 years of total professional experience (minimum 5 years).
- 3–5 years of experience in Data Engineering or a related discipline.
- Experience in financial services or product development environments is a strong plus.
- Expertise in Python and SQL with strong data manipulation and analysis skills.
- Proficiency with Databricks/PySpark and dbt for data warehousing and transformation.
- Hands-on experience with workflow orchestration tools (e.g., Airflow, Temporal).
- Experience working with LLMs, prompt engineering, RAG frameworks, and vector databases.
- Knowledge of machine learning fundamentals, feature engineering, and knowledge graphs is an advantage.
- Strong experience in designing and implementing complex data systems from scratch.
- Ability to handle large-scale data projects including ETL, data cleaning, and information retrieval.
- Strong communication skills (verbal and written).
- Comfortable working the shift from 3:00 PM IST to 12:00 AM IST.