
Lead Azure Databricks Data Engineer
Ryan Specialty
full-time
Posted on:
Location Type: Office
Location: Carmel • Illinois • United States
Visit company websiteExplore more
Salary
💰 $112,000 - $140,000 per year
Job Level
About the role
- Provide technical leadership in data architecture and engineering, collaborating with solution stakeholders to align data solutions with business objectives
- Mentor less experienced data engineers and team members, fostering a collaborative and innovative work environment
- Explore and implement new technologies and practices in data engineering to continually improve our data processes and infrastructure
- Manage and enhance complex SSIS packages for data integration and ETL processes, along with SSRS reports for business intelligence
- Design, build, and maintain scalable and efficient data pipelines using Azure Data Factory, implementing both batch and real-time data integration processes
- Develop and optimize Databricks notebooks for big data processing and analytics in Azure Databricks environments
- Implement and manage data storage solutions using Delta Parquet files in Azure Data Lake Storage, optimizing for query performance and cost-efficiency
- Integrate and configure Profisee Master Data Management tools within Azure, focusing on data governance and quality
- Set up and manage Change Data Capture processes for real-time data synchronization in Azure data platforms
- Implement data cataloging, classification, and lineage using Azure Purview, developing data governance policies, and ensuring compliance
- Enhance and support Tableau reporting solutions, developing complex dashboards, and integrating with Azure data services
- Drive automation and process optimization using Azure services and scripting and implement CI/CD pipelines using Azure DevOps
- Lead SQL Server database optimization and performance tuning efforts
- Lead, Coach and Mentor junior staff on projects
Requirements
- Bachelor’s degree required, Master’s degree preferred; Computer science, Information Technology or Engineering is preferred
- 10+ years of experience in Data Engineering with a Bachelor’s degree, or 8 years of experience with a Master’s degree is preferred
- 6+ years of experience with Databricks
- Subject Matter Expert in SQL Server Management, Azure Data Factory Development, Databricks Engineering, Delta Parquet File Management, and SQL Server Integration Services
- Proficiency in Profisee MDM Integration, Change Data Capture implementation, and data quality and governance with Azure Purview
- Proven experience in Tableau and BI Reporting, and strong skills in Automation and Process Skills
Benefits
- Paid time off for company holidays, vacation, sick and personal days
- Paid parental leave
- Mental health services
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills & Tools
data architecturedata engineeringETL processesdata pipelinesbig data processingdata storage solutionsdata governancedata catalogingSQL Server optimizationautomation
Soft Skills
technical leadershipmentoringcollaborationinnovationcoaching