Senior Data Engineer
Quest Analytics, LLC
AzureCloudETLMongoDBNoSQLNumpyPandasPythonScalaSparkSQL
Analyst/Consultant - Data Engineer, Azure
Blue Altair
full-time
Posted on:
Origin: • 🇮🇳 India
Visit company websiteJob Level
Mid-LevelSenior
Tech Stack
AzureCassandraETLMongoDBNoSQLPySparkPythonSQL
About the role
- Design and implement data pipelines to deliver data to the Business Intelligence (BI) team
- Create and manage Azure Data Factory pipelines to extract data and load into Azure Blob Storage (Bronze layer)
- Develop Azure Databricks pipelines to process and transform data from bronze to silver layer, creating intermediate tables
- Build and maintain pipelines to transform silver layer data into Gold layer fact and dimension tables for BI analysis
- Monitor, troubleshoot, and optimize all data pipelines to ensure smooth, efficient operation
- Integrate data from databases, APIs, and third-party systems and develop ETL solutions to data warehouses or data lakes
- Manage and optimize relational and NoSQL databases; implement and maintain data models and database schemas
- Implement and enforce data governance policies and ensure data security and regulatory compliance
Requirements
- Title: Analyst/Consultant - Data Engineer
- Experience: 3-4 years (minimum 36 months)
- Location: Pune
- Data Pipeline Development experience: design and implement data pipelines for BI teams
- Azure Data Factory (ADF): expertise in designing and managing data workflows
- Azure Databricks: hands-on experience for data engineering and pipeline development
- Python: proficiency for scripting and automation tasks
- PySpark: experience for large-scale data processing and transformations
- Data Integration: integrate data from databases, APIs, third-party systems; ETL to data warehouses or data lakes
- Database Management: manage and optimize relational (e.g., SQL) and NoSQL (e.g., MongoDB, Cassandra) databases; implement and maintain data models and schemas
- Data Governance and Security: implement and enforce data governance policies and ensure data security and compliance
- Monitoring and Optimization: monitor, troubleshoot, and optimize data pipelines
- Soft skills: excellent communication and teamwork abilities; ability to manage multiple tasks and projects; proactive, self-motivated, continuous learning mindset
