Data Engineer, Palantir, Spark, PySpark, Python

CloudScouts

Technical Lead in Data Engineering driving scalable data solutions for a large retail enterprise. Requires expertise in Palantir, Spark, PySpark, SQL, and Python with leadership experience.

Posted 4/27/2026full-timeAlabama, Alaska, Arizona, California, Colorado, Connecticut, Florida, Hawaii, Idaho, Illinois, Iowa, Kansas, Kentucky, Louisiana, Maine, Maryland, Massachusetts, Minnesota, Mississippi, Missouri, Montana, Nevada, New Hampshire, New Jersey, New Mexico, New York, North Carolina, North Dakota, Ohio, Oklahoma, Oregon, Pennsylvania, Rhode Island, South Carolina, South Dakota, Tennessee, Texas, Utah, Vermont, Virginia, Washington, West Virginia, Wisconsin, Wyoming • 🇺🇸 United StatesSeniorLeadWebsite

Tech Stack

Tools & technologies

ApacheDistributed SystemsETLPySparkPythonSparkSQL

About the role

Key responsibilities & impact

Lead the design and implementation of scalable data platforms and pipelines using PySpark, Spark, and Python
Drive adoption and best practices for Palantir Foundry (data pipelines, ontology, workflows, and operational applications)
Architect and optimize high-performance data processing solutions for large-scale datasets
Provide technical leadership and mentorship to data engineers, ensuring code quality and best practices
Collaborate with cross-functional teams (business, analytics, data science) to translate requirements into scalable solutions
Design robust data models, ETL/ELT frameworks, and data integration strategies
Ensure data quality, governance, security, and compliance across enterprise data platforms
Lead performance tuning, troubleshooting, and optimization of data pipelines
Drive CI/CD implementation, code reviews, and release management
Stay current with emerging data engineering technologies and recommend improvements

Requirements

What you’ll need

Bachelor’s or Master’s degree in Computer Science, Engineering, or related field
8+ years of experience in data engineering, with at least 2–3 years in a technical leadership role
Strong hands-on experience with: Python & PySpark, Apache Spark, Advanced SQL, Experience with Palantir Foundry or similar modern data platforms, Deep understanding of data engineering principles (ETL/ELT, data modeling, distributed systems)
Experience designing and managing large-scale data architectures
Strong leadership, communication, and stakeholder management skills

Benefits

Comp & perks

Flexible work from home options

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

PythonPySparkApache SparkAdvanced SQLETLELTdata modelingdata integrationperformance tuningdata processing

Soft Skills

leadershipcommunicationstakeholder managementmentorshipcollaboration

Certifications

Bachelor’s degreeMaster’s degree