Senior Data Engineer
The Muse
ApacheAWSAzureCloudETLPySparkPythonScalaSparkSQL
Data Engineer - Databricks
DXC Technology
full-time
Posted on:
Location Type: Hybrid
Location: Newcastle • 🇬🇧 United Kingdom
Visit company websiteJob Level
JuniorMid-Level
Tech Stack
AirflowApacheETLPandasPySpark
About the role
- Build and maintain scalable data pipelines using Databricks
- Manage and optimize Databricks Workspace, Clusters, Jobs, Repos, and Delta Live Tables
- Develop modular ETL workflows using dbt and orchestrate them with Apache Airflow
- Write advanced PySpark code, including UDFs and Pandas UDFs, for efficient data processing
- Ensure data quality, reliability, and performance across all systems
- Collaborate with data scientists, analysts, and engineers to deliver data solutions that support business goals
- Contribute to the mentoring and development of junior team members
- Support senior team members in identifying and addressing data science opportunities
- Travel within the UK when required depending on project requirements
Requirements
- Sole UK nationals required and willing to undergo SC clearance
- Willingness to travel within the UK when required
- Proven experience with Databricks and its ecosystem
- Strong proficiency in PySpark, including UDFs and Pandas UDFs
- Hands-on experience with dbt and Apache Airflow
- Databricks Certified Data Engineer Associate certification or willingness to achieve one
- Excellent problem-solving skills and a passion for data and AI
- Bachelor's degree in a relevant field or equivalent combination of education and experience
- Typically 5+ years of relevant work experience in industry, with a minimum of 2+ years in a similar role
- Proficiencies in data cleansing, exploratory data analysis, and data visualization
- Continuous learner who stays abreast with industry knowledge and technology
Benefits
- Competitive + Benefits
- Hybrid options available
- Access to continuous learning and development opportunities
- Flexible working arrangements
- Inclusive and supportive company culture
- Work with a forward-thinking team on impactful AI projects
ATS Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard skills
DatabricksPySparkUDFsPandas UDFsdbtApache AirflowETL workflowsdata cleansingexploratory data analysisdata visualization
Soft skills
problem-solvingcollaborationmentoringcommunicationcontinuous learning
Certifications
Databricks Certified Data Engineer Associate
Similar jobs on JobTailor
Power BI / Data Warehouse Engineer
MULTI SEAL Tire Sealants
AirflowApacheCloudETLPandasPythonScikit-LearnSQLTableau
Senior Data Developer
CI&T
AirflowApacheAzureCloudETLPythonSparkSQL
Senior Data and BI Engineer
Alex Staff Agency
AirflowAWSCloudDynamoDBETLPythonSparkSQL
Senior Data and BI Engineer
Alex Staff Agency
AirflowAWSCloudDynamoDBETLPythonSparkSQL