Senior Data Engineer, Databricks

State Street

Sr. Databricks Data Engineer building Databricks and Azure data pipelines for State Street’s financial technology platforms.

Posted 8/20/2026full-timeQuincy • Massachusetts, New Jersey • 🇺🇸 United StatesSenior💰 $120,000 - $202,500 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in designing and implementing end-to-end data pipelines using Azure, Databricks, and cloud-based infrastructure, with a strong focus on ETL processes and data quality. Proficient in application architecture, micro-services, and agile methodologies, with a proven ability to collaborate effectively within teams.

Highest-signal resume keywords

Azure Cloud ComputingDatabricks SQLETL ProcessesData Pipeline MonitoringAgile Development Methodology

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Data Pipeline DevelopmentDatabricksApache SparkScalaSQLData Quality ChecksShell ScriptingUNIXVersion Control (Git)Data Analysis

Soft Skills

Effective CommunicationTeam CollaborationProblem SolvingSelf-MotivationPositive Attitude

Tools & Technologies

Azure DevOpsKubernetesDockerCosmos DBSnowflakeTOADDelta LakeADOHarness

Certifications & Qualifications

Bachelor’s Degree in Computer or IT-related Subject

Industry Keywords

Big DataFinancial IndustryAgile MethodologyMicro-Services Architecture

Tech Stack

Tools & technologies

ApacheAzureCloudDockerETLKubernetesScalaShell ScriptingSparkSQLUnix

About the role

Key responsibilities & impact

Design, build, and test end-to-end data pipelines, including streaming, events, and batch data ingestion, on cloud-based infrastructure using Azure and Cosmos DB
Develop and implement ETL processes for data ingestion, transformation, and loading into data lakes
Work extensively on Databricks and data warehousing concepts
Design and develop custom high-throughput and configurable frameworks and libraries
Drive change through collaboration, influence, and demonstration of POCs
Manage all aspects of the software development lifecycle, including design, coding, integration testing, deployment, and documentation
Work collaboratively within an agile project team
Follow best practices and coding standards
Grow personal skillset

Requirements

What you’ll need

Extensive technical, people and process skillsets
Strong background in application architecture and development
Very good understanding of leading IT technologies
Experience in micro-services architecture and understanding of Cloud Computing; Azure preferred
Proficient working knowledge of Databricks, Delta Lake, and Apache Spark
Hands-on development experience in Databricks SQL and Scala
Strong understanding of the Databricks platform, including clusters, jobs, and other resources
Ability to monitor and troubleshoot data pipelines
Experience implementing data quality checks and validations
Experience performing data analysis and data exploration
Strong hands-on experience troubleshooting DevOps pipelines, including ADO and Harness
Experience in a multi-developer environment using version control such as Git
Experience handling semi-structured data including Avro, JSON, and XML
Basic knowledge of shell scripting, UNIX, TOAD, and SQL Developer
Bachelor’s Degree level qualification in a computer or IT-related subject
10+ years of overall Big Data data pipeline experience
5+ years of hands-on Databricks experience
5+ years of cloud-based development experience, including Azure Services, Azure DevOps, Kubernetes, and Docker
Experience with Snowflake is a plus
Effective professional written and oral communication
Team player with a positive attitude, enthusiasm, initiative, and self-motivation
Ability to multitask; energetic, fast learner, and problem solver
Experience working in the financial industry
Experience with agile development methodology

Benefits

Comp & perks

Retirement savings plan (401K) with company match
Insurance coverage including basic life, medical, dental, vision, long-term disability, and other optional additional coverages
Paid time off including vacation, sick leave, short term disability, and family care responsibilities
Employee Assistance Program
Incentive compensation including eligibility for annual performance-based awards
Certain tax advantaged savings plans
Inclusive development opportunities
Flexible work-life support
Paid volunteer days
Vibrant employee networks