Design and implement complex Spark data logic, focusing on performance optimization, data volume tuning, and robust execution.
Own the architectural design of graph build pipelines, ensuring they are scalable, automated, and highly resilient.
Plan and oversee the strategic re-architecture of data pipelines to meet evolving business needs and scale.
Optimize infrastructure-as-code and schema designs to reduce cloud costs and improve pipeline latency.
Act as a technical consultant for the team, fostering a collaborative and engineer-led approach to design decisions.
Support the development of the engineering team through code reviews, design docs, and architectural best practices.
Ensure the accuracy of mission-critical data outputs.

Requirements

8+ years of experience in the big data space, with a proven track record of implementing large-scale features and leading process redesigns.
Expert-level mastery of Apache Spark for large-scale data processing.
Strong experience with orchestration tools (Airflow) and cloud computing environments.
Hands-on experience architecting and managing data flows into databases such as Elasticsearch, Memgraph, and Cassandra.
Demonstrated ability in system architecture, including Infrastructure as Code (IaC) and schema design.
A "builder" mindset with experience evolving and improving existing architectures to meet new scale requirements.

Benefits

100% fully paid medical, vision, and dental for employees and their dependents
Generous time off; we observe all US federal holidays, close our office for a winter break (12/24-12/31), in addition to granting 18 PTO days and 10 sick days
Outstanding compensation package; competitive commissions for revenue roles and quarterly bonuses for non-revenue positions
A strong commitment to diversity, equity, and inclusion
Eligibility to participate in additional benefits such as 401k match up to 5%, 100% paid life insurance (up to $100,000 coverage),, and parental leave
A collaborative and positive culture - your team will be as smart and driven as you
Limitless growth and learning opportunities

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard skills

Apache Sparkdata processingInfrastructure as Codeschema designdata volume tuningperformance optimizationsystem architecturedata flowsorchestration toolscloud computing

Soft skills

collaborative approachtechnical consultingcode reviewsdesign documentationarchitectural best practicesbuilder mindset