Databricks Data Engineer
Sequoia Connect
ApacheAWSAzureCloudJavaPythonScalaSparkSQL
Salary
💰 £80,000 - £95,000 per year
Job Level
Senior
Tech Stack
ApacheAWSAzureCloudDockerGoogle Cloud PlatformHadoopSpark
About the role
- Define, shape and perfect data strategies in central and local government
- Help public sector teams understand the value of their data, and make the most of it
- Establish yourself as a trusted advisor in data driven approaches using public cloud services like AWS, Azure and GCP
- Contribute to recruitment efforts and take on line management responsibilities
- Implement efficient data pipelines & storage
- Join a data team on its mission to get data knowledge and skills out of silos and embedded into delivery teams
Requirements
- Proficiency in Git (inc. Github Actions) and able to explain the benefits of different branch strategies
- Strong experience in IaC and able to guide how one could deploy infrastructure into different environments
- Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop
- Good understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes)
- Ability to create data pipelines on a cloud environment and integrate error handling within these pipelines
- You understand how to create reusable libraries to encourage uniformity or approach across multiple data pipelines
- Able to document and present end-to-end diagrams to explain a data processing system on a cloud environment
- Some knowledge of how you would present diagrams (C4, UML, etc.)
- Enthusiasm for learning and self-development
- You have experience of working on agile delivery-lead projects and can apply agile practices such as Scrum, XP, Kanban
- Can own the cloud infrastructure underpinning data systems through a DevOps approach
- Design and implement efficient data transformation processes at scale, both in batch and streaming use cases
- You are a skilled Data Engineer who has delivered data platforms
- Knowledge of SOLID, DRY and TDD principles and how to practically implement these into a project
- You can demonstrate a commercial mindset when on projects to grow accounts organically with senior stakeholders
- You have the experience to improve resilience into a project by checking for software vulnerabilities and implement appropriate testing strategies (unit integration, data quality, etc)
- You are skilled at offering guidance on how one would implement a robust DevOps approach in a data project
- You can comfortably talk about tools needed such as DataOps in areas such as orchestration, data integration and data analytics
- Desirable: Working at a technology consultancy
- Desirable: Working with Docker and virtual environments as part of the development and CI/CD process
- Desirable: Working with senior stakeholders to gather requirements and keep them engaged with
- Desirable: Experience in working with a team of engineers using a variety of techniques such as pair programming or mob programming
- Desirable: Working with data scientists to productionise advanced data deliverables, such as machine learning models
- Desirable: Working knowledge of statistics
- Desirable: Working with multidisciplinary digital and technology teams
- Desirable: Working within the public sector
Similar jobs on JobTailor
Senior Data Engineer
CI&T
ApacheAWSAzureCloudGoogle Cloud PlatformPythonSparkSQLTerraformVault
Senior Application Support Developer
<Undefined>
ApacheAWSAzureDockerHibernateJavaJUnitKubernetesLinuxMariaDBMicroservicesMockito+6 more
Resident Solutions Architect, Public Sector
Databricks
ApacheAWSAzureBootstrapCloudGoogle Cloud PlatformPythonScalaSpark
Senior Solution Architect
CYOS Solutions
ApacheAWSAzureCloudHadoopKafkaMicroservicesSpark