FREE ACCESS
5,000–10,000 jobs/day
See all jobs on JobTailor
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Tech Stack
Tools & technologiesAWSKafkaLinuxPySparkPythonSQLTerraformUnity
About the role
Key responsibilities & impact- Lead the design and implementation of complex data ingestion pipelines using PySpark, Databricks Intelligence Platform, and AWS services (S3, IAM roles and policies), including Kafka/streaming architectures, schema evolution, and Delta Share, ensuring alignment with enterprise architecture standards.
- Serve as a subject matter expert (SME) for data engineering, providing technical guidance and mentorship to engineers across the team and representing data engineering in cross-functional discussions.
- Take ownership of platform reliability and performance, proactively identifying optimization opportunities across data ingestion, transformation, and storage layers.
- Troubleshoot complex pipeline failures across multi-system dependencies.
- Lead cross-functional technical initiatives, collaborating with data scientists, analysts, DevOps engineers, client teams (upstream data providers and downstream consumers), and stakeholders to deliver integrated solutions that meet program objectives.
- Contribute to the data platform technical vision and architecture, making strategic decisions on tooling, frameworks, and design patterns that shape the program's long-term data engineering direction.
- Define and enforce engineering standards, including code quality, testing practices, CI/CD processes, and infrastructure as code patterns using Terraform.
- Evaluation of new technologies and approaches, presenting proof of concepts and technical roadmaps to leadership.
- Author and maintain architectural documentation, technical decision records, and platform runbooks that enable team autonomy and operational excellence.
- Other relevant duties as assigned and qualified/trained to perform.
Requirements
What you’ll need- Bachelor’s degree.
- Minimum of 5 years of experience in Data Engineering.
- Demonstrated experience serving as a technical SME and leading cross-functional engineering initiatives.
- Proficiency in Python for data engineering tasks at scale.
- Deep expertise with Databricks Intelligence Platform (Notebooks, Lakeflow Jobs, Unity Catalog, Delta Share) and the ability to define platform standards and best practices for the team.
- Experience with AWS services such as S3 and IAM roles and policies.
- Strong knowledge of data pipeline design and implementation, including data transformation techniques, data modeling, data storage optimization, and data security best practices.
- Proficiency in using version control systems like Git for managing code repositories and collaborating with team members on Agile projects.
- Familiarity with Terraform or other infrastructure as code tools for automating infrastructure deployment and configuration management.
- Experience working on Linux environments for data engineering projects, including accessing containers remotely, installing packages, managing files, services, and processes.
- Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
- Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
- Strong problem-solving skills, ability to work independently and as part of a team, and excellent verbal and written communication skills.
- Comfortable working in a highly collaborative environment with strong attention to detail and a commitment to delivering high-quality software.
- Ability to obtain and maintain a Public Trust security clearance.
Benefits
Comp & perks- Flex hours
- 401K with matching incentive
- Parental Leave
- Medical/dental/vision benefits
- Flex Spending Account
- Company provided short-term disability and life insurance
- Commuter benefits
- Paid Time Off (PTO)
- 11 Paid holidays
ATS Keywords
✓ Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills & Tools
Python ProgrammingData Pipeline DesignSQL Query AuthoringData Transformation TechniquesData ModelingData Storage OptimizationRoot Cause AnalysisVersion Control (Git)Infrastructure as Code (Terraform)Linux Environment Management
Soft Skills
Problem-SolvingCollaborationAttention to DetailVerbal CommunicationWritten Communication
Certifications
Public Trust Security Clearance