Ryan Specialty

Lead Azure Databricks Data Engineer

Ryan Specialty

full-time

Posted on:

Location Type: Office

Location: CarmelIllinoisUnited States

Visit company website

Explore more

AI Apply
Apply

Salary

💰 $112,000 - $140,000 per year

Job Level

About the role

  • Provide technical leadership in data architecture and engineering, collaborating with solution stakeholders to align data solutions with business objectives
  • Mentor less experienced data engineers and team members, fostering a collaborative and innovative work environment
  • Explore and implement new technologies and practices in data engineering to continually improve our data processes and infrastructure
  • Manage and enhance complex SSIS packages for data integration and ETL processes, along with SSRS reports for business intelligence
  • Design, build, and maintain scalable and efficient data pipelines using Azure Data Factory, implementing both batch and real-time data integration processes
  • Develop and optimize Databricks notebooks for big data processing and analytics in Azure Databricks environments
  • Implement and manage data storage solutions using Delta Parquet files in Azure Data Lake Storage, optimizing for query performance and cost-efficiency
  • Integrate and configure Profisee Master Data Management tools within Azure, focusing on data governance and quality
  • Set up and manage Change Data Capture processes for real-time data synchronization in Azure data platforms
  • Implement data cataloging, classification, and lineage using Azure Purview, developing data governance policies, and ensuring compliance
  • Enhance and support Tableau reporting solutions, developing complex dashboards, and integrating with Azure data services
  • Drive automation and process optimization using Azure services and scripting and implement CI/CD pipelines using Azure DevOps
  • Lead SQL Server database optimization and performance tuning efforts
  • Lead, Coach and Mentor junior staff on projects

Requirements

  • Bachelor’s degree required, Master’s degree preferred; Computer science, Information Technology or Engineering is preferred
  • 10+ years of experience in Data Engineering with a Bachelor’s degree, or 8 years of experience with a Master’s degree is preferred
  • 6+ years of experience with Databricks
  • Subject Matter Expert in SQL Server Management, Azure Data Factory Development, Databricks Engineering, Delta Parquet File Management, and SQL Server Integration Services
  • Proficiency in Profisee MDM Integration, Change Data Capture implementation, and data quality and governance with Azure Purview
  • Proven experience in Tableau and BI Reporting, and strong skills in Automation and Process Skills
Benefits
  • Paid time off for company holidays, vacation, sick and personal days
  • Paid parental leave
  • Mental health services
Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools
data architecturedata engineeringETL processesdata pipelinesbig data processingdata storage solutionsdata governancedata catalogingSQL Server optimizationautomation
Soft Skills
technical leadershipmentoringcollaborationinnovationcoaching