Data Platform Engineer
IncWorx Consulting
AzureETLPythonSQLSSIS
Principal Platform Engineer – Data Ops Engineer
DIRECTV
full-time
Posted on:
Location Type: Remote
Location: Remote • California • 🇺🇸 United States
Visit company websiteSalary
💰 $116,375 - $211,255 per year
Job Level
Lead
Tech Stack
AirflowApacheAWSJavaScriptJenkinsJMeterPySparkPythonShell ScriptingSparkSplunk
About the role
- Define the long-term orchestration strategy and architectural standards for workflow management across Snowflake, Databricks, and AWS
- Lead the design, implementation, and optimization of complex workflows using Apache Airflow and related tools
- Architect and guide the development of reusable automation frameworks in Python, Spark, and Shell that streamline data workflows
- Define and maintain enterprise-wide CI/CD standards for data pipelines
- Lead performance audits and capacity planning efforts across Snowflake, Databricks, and orchestrated workflows
- Oversee complex incident resolution and implement systemic preventive measures
Requirements
- 4 – 5 years required, 10+ years preferred of overall software engineering experience , including 10+ years preferred as a Big Data Architect
- 4 – 5 years required, 8+ years preferred of hands-on programming experience with Python, PySpark, JavaScript, and Shell scripting
- 5+ years of experience designing and implementing configuration-driven frameworks in PySpark on Databricks
- 4 – 5 years required, 8+ years preferred of experience in CI/CD pipeline development and automation using GitLab, Jenkins, and Databricks REST APIs
- 4 – 5 years required, 8+ years preferred of deep expertise in Snowflake, Databricks, and AWS
- 7+ years of experience in performance monitoring and observability using tools like SonarQube, JMeter, Splunk, Datadog, and AWS CloudWatch
- 7+ years of experience in Tier 2/3 support roles
- 4+ years of experience with dbt
- Deep expertise in Apache Airflow and orchestration technologies
- Strong analytical and architectural skills to design, optimize, and troubleshoot complex data pipelines
Benefits
- Health insurance
- 401(k) matching
- Flexible work hours
- Paid time off
- Remote work options
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard skills
PythonPySparkJavaScriptShell scriptingApache AirflowCI/CDdbtworkflow managementdata pipelinesperformance monitoring
Soft skills
analytical skillsarchitectural skillstroubleshootingleadershipincident resolutioncapacity planningoptimization
