Design and ship platform services for ingestion, transformation, orchestration, and metadata (e.g., service‑backed interfaces for Dagster/Airflow, lineage, quality, and data contracts).
Build execution & scheduling capabilities for Spark/SQL jobs (queuing, prioritization, retries, resource isolation on EMR/EKS/Databricks), focusing on throughput and developer experience.
Implement lakehouse features (Delta/Iceberg): schema evolution, partitioning, compaction, vacuum, snapshotting, ACID guarantees, and table‑format governance
Optimize Snowflake and other warehouses: cost controls, query profiling/pruning, workload isolation, RBAC; expose safe self‑service patterns.
Deliver SDKs, CLIs, and templates that standardize how teams build reliable data products; enable CI/CD for data and contract testing.
Work across AWS (S3, EMR/EKS, Glue/Athena, Lambda, Kinesis/MSK) with IaC (Terraform) and strong observability (Datadog/CloudWatch).

Requirements

8+ years building backend/platform software with Python/Scala/Java and strong SQL; proven track record designing distributed systems.
Deep experience with Spark (Databricks or EMR/EKS) and AWS data services; solid grasp of scheduler/executor behavior and performance tradeoffs.
Hands‑on data warehouse optimization (Snowflake ideal; others welcome).
Experience building platform APIs/SDKs that other engineers adopt; excellent collaboration and technical leadership.

Benefits

Flexible, hybrid work environment at our convenient Soho location
Unlimited Vacation
100% paid employee health benefit options (including medical, dental, and vision)
Commuter Benefits
401(k) with employer funded match
Corporate wellness program with Wellhub
Sabbatical leave (for employees with 5+ years of service)
Competitive paid parental leave and fertility/family planning reimbursement
Cell phone reimbursement
Catered lunch everyday along with beverages and snacks
Employee Resource Groups and ZocClubs to promote shared community and belonging
Great Place to Work Certified

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

PythonScalaJavaSQLSparkDatabricksSnowflakeTerraformACIDdata contracts

Soft Skills

collaborationtechnical leadership