Senior Data Engineer

CDW

Senior Data Engineer developing scalable data solutions on the Databricks platform at CDW. Collaborating with architects and data science teams to deliver high-quality data products.

Posted 5/17/2026full-timeRemote • Illinois • 🇺🇸 United StatesSeniorWebsite

Tech Stack

Tools & technologies

AzureETLPySparkPythonPyTorchScikit-LearnSparkSQLTensorflowUnity

About the role

Key responsibilities & impact

Build and maintain scalable data pipelines, ETL and ELT processes, and data models within the Databricks platform.
Design, develop, and deploy data and AI solutions using Databricks, Spark, Delta Lake, and related technologies.
Develop batch and streaming pipelines using tools such as Databricks Workflows and Azure Data Factory.
Design logical data flow diagrams and normalized schemas, implementing Lakehouse patterns such as the Medallion Architecture (Bronze, Silver, Gold layers).
Ensure data quality, integrity, security, and governance throughout the data lifecycle, including use of Unity Catalog.
Optimize Spark jobs and data transformations through effective partitioning, caching, and join strategies.
Monitor pipeline execution, identify failures, and troubleshoot complex data processing issues.
Collaborate with data architects, analysts, data scientists, and business stakeholders to understand requirements and deliver solutions.
Support documentation of data processes, standards, and data flows.

Requirements

What you’ll need

5 Years of experience designing, developing, and deploying data solutions on the Databricks platform.
Proficiency in Python, including PySpark, and SQL.
Hands-on experience with Spark, Delta Lake, and Lakehouse architectures.
Experience implementing data quality, governance, and security practices across data pipelines.
Familiarity with machine learning concepts, tools, and libraries such as TensorFlow, PyTorch, Scikit-learn, and MLflow is a plus.
Experience configuring and integrating external AI models and working with AI governance and monitoring tools is a plus.
Experience with asynchronous programming patterns in Python for building scalable data or AI workloads is a plus.
Strong problem-solving, collaboration, and communication skills.

Benefits

Comp & perks

Health insurance
401(k) matching
Paid time off
Professional development

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

PythonPySparkSQLSparkDelta LakeETLELTdata modelingasynchronous programmingmachine learning

Soft Skills

problem-solvingcollaborationcommunication