Senior Staff Data Engineer, Databricks

Shield AI

Senior Staff Data Engineer managing enterprise data lakehouse on Databricks for Shield AI. Focusing on scalable ingestion and reliable data processing across multiple business domains.

Posted 7/30/2026full-timeRemote • 🇺🇸 United StatesSenior💰 $140,000 - $210,000 per yearWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates extensive experience in building and operating data pipelines within a Databricks lakehouse environment, with a strong focus on data governance, quality controls, and compliance requirements. Proficient in designing scalable ingestion and transformation patterns while collaborating effectively with cross-functional teams.

Highest-signal resume keywords

Databricks ExperienceDelta LakeCI/CD Pipeline DevelopmentSQL SkillsData Classification and Compliance

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

Data EngineeringData Pipeline DevelopmentData ModelingBatch IngestionStreaming IngestionMetadata CaptureData Quality ControlsTransformation PatternsSpark with PySparkSpark SQL

Soft Skills

CollaborationProblem-SolvingBusiness Understanding

Tools & Technologies

Databricks WorkflowsUnity CatalogVersion Control Systems

Industry Keywords

Data GovernanceRegulatory ComplianceData SecurityData RetentionMedallion Architecture

Tech Stack

Tools & technologies

PySparkSparkSQLUnity

About the role

Key responsibilities & impact

The Senior Staff Data Engineer will help build and operate the enterprise lakehouse on Databricks, creating the governed data foundation that supports multiple business domains and downstream analytics.
Responsible for scalable ingestion, reliable data processing, and strong technical controls across the Bronze and Silver layers of the medallion architecture.
Design and build ingestion pipelines from enterprise source systems into the Databricks lakehouse using Delta Lake.
Own Bronze-layer ingestion, including raw landing patterns, metadata capture, load traceability, and recoverable ingestion design.
Build Silver-layer pipelines for cleansing, standardization, deduplication, conformance, and quality enforcement.
Define and evolve reusable ingestion and transformation patterns, templates, and engineering standards.
Implement and maintain Databricks platform constructs needed for secure delivery.
Build and maintain CI/CD pipelines for data platform assets.
Apply data classification, segregation, and handling requirements within the pipeline design.
Build data quality controls that reflect actual business meaning, record integrity, completeness, and expected domain behavior.
Maintain documentation for source objects, ingestion logic, applied transformations, data quality rules, and known limitations.
Partner with the Analytics Engineer and domain teams to ensure Silver-layer data is reliable, well-governed, and suitable for trusted Gold-layer modeling.
Collaborate with domain engineering teams to align on ownership boundaries, onboarding patterns, data contracts, and support expectations as new domains are enabled onto the platform.

Requirements

What you’ll need

12+ years of data engineering experience, including hands-on ownership of production data pipelines.
Strong Databricks experience, including Delta Lake, Databricks Workflows or Jobs, and Spark with PySpark and/or Spark SQL.
Working knowledge of Unity Catalog, including catalogs, schemas, tables, lineage, and access control concepts.
Experience with batch, CDC, and/or streaming ingestion patterns and the operational trade-offs associated with each.
Experience with CI/CD and deployment automation for data pipelines and platform assets, including version control, testing, and controlled promotion across environments.
Strong SQL skills and solid grounding in data modeling fundamentals, even if dimensional modeling is not the primary responsibility of this role.
Demonstrated ability to understand the business and regulatory context of the data being processed, not just the mechanics of pipeline development.
Experience applying data classification, segregation, security, retention, or compliance requirements in data engineering workflows within a regulated or security-sensitive environment.
Ability to design pipelines with awareness of the actual data domains involved, including sensitivity, ownership, permitted use, and downstream impact.
Comfort operating in a fast-moving platform build where patterns are still being established and engineers are expected to shape standards, not just follow them.

Benefits

Comp & perks

Pay within range listed + Bonus + Benefits + Equity
Temporary benefits package (applicable after 60 days of employment)