Explore more
About the role
- Data Pipeline Development: Design, build, and optimize robust ETL/ELT pipelines using Azure Databricks, Spark, and SQL.
- Data Processing & Transformation: Utilize Python, PySpark, and SQL to clean, transform, and aggregate complex data for analytics.
- Azure Data Lake Management: Manage and optimize data storage and retrieval in Azure Data Lake Storage (ADLS) Gen2.
- Delta Lake Implementation: Implement Delta Lake for ACID compliance, data versioning, and high-performance data lake operations.
- Integration with Azure Services: Integrate Databricks with Azure Data Factory for orchestration, Azure Synapse Analytics for warehousing, and Azure Key Vault for security.
- Performance Optimization: Monitor, troubleshoot, and optimize Databricks clusters and spark jobs to manage costs and performance.
- Data Security & Governance: Implement Role-Based Access Control (RBAC), data encryption, and data lineage tracking.
- Requirements Collaboration: Work with data scientists and analysts to support machine learning models and business intelligence (BI) reporting.
Requirements
- Databricks & Spark: Strong proficiency in PySpark, Spark SQL, and Databricks Jobs.
- Azure Infrastructure: Familiarity with Azure Data Factory, ADLS, and Synapse Analytics.
- Data Modelling: Experience in building data models (e.g., Star Schema, Snowflake).
- Programming: Proficient in Python and SQL.
- DevOps: Experience with Git and CI/CD tools.
Benefits
- Diversity Inclusion: At Exavalu, we are committed to building a diverse and inclusive workforce.
- We nurture a culture that embraces all individuals and promotes diverse perspectives, where you can make an impact and grow your career.
- Exavalu also promotes flexibility depending on the needs of employees, customers and the business. It might be part-time work, working outside normal 9-5 business hours or working remotely.
- We also have a welcome back program to help people get back to mainstream after a long break due to health or family reasons.
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills & Tools
ETLELTAzure DatabricksSparkSQLPythonPySparkDelta LakeData ModellingGit
Soft Skills
collaborationtroubleshootingperformance optimization