MediaRadar, Inc.

Data Engineering Lead

MediaRadar, Inc.

full-time

Posted on:

Location Type: Remote

Location: United States

Visit company website

Explore more

✨ AI Apply
Apply

Job Level

Senior

Tech Stack

AirflowApacheAzureETLKubernetesPostgresPySparkSparkSQLTerraform

About the role

  • Design and supervise the implementation of comprehensive data pipelines utilizing Azure Databricks and PySpark.
  • Direct a team of data engineers, performing code reviews, offering technical expertise, and cultivating a culture of ongoing learning.
  • Develop high-performance schemas in PostgreSQL and refine complex SQL queries for large datasets.
  • Establish and apply optimal practices for data ingestion, transformation, and storage (Delta Lake/Lakehouse patterns).
  • Collaborate closely with Data Analysts, Architects, and Product Managers to convert business requirements into technical specifications.
  • Promote the implementation of CI/CD, unit testing, and automated monitoring to achieve 99.9% data reliability.
  • Ensure data quality, governance, and compliance through validation, documentation, and secure practices.
  • Continuously improve data systems for enhanced performance, reliability, and scalability.
  • Effectively engage within an agile, cross-functional project team.

Requirements

  • Azure Databricks:
  • ○ Expert-level experience managing workspaces, clusters, and job scheduling.
  • ○ Solid understanding of data lakehouse architectures and Delta Lake.
  • ○ Proven experience in Performance Tuning, Spark Optimization and Cost Reduction.
  • PySpark: Advanced proficiency in Spark DataFrame APIs and Spark SQL for large-scale data processing involving various data formats.
  • SQL Mastery: Exceptional ability to write, tune, and troubleshoot complex queries.
  • PostgreSQL: Hands-on experience with relational database design, indexing, and performance optimization.
  • ETL/ELT Frameworks: Proven track record of building scalable data pipelines from scratch.
  • Workflow Orchestration: Experience with Apache Airflow for managing complex task dependencies.
  • Containerization: Familiarity with Azure Kubernetes Service (AKS) for deploying containerized data services.
  • Infrastructure as Code (IaC): Knowledge of Terraform or Bicep for managing Azure resources.
  • 10+ years of experience in Data Engineering or Software Engineering.
  • 3+ years as a formal technical Lead managing an agile team and implementing E2E solutions.
  • Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
  • Strong communication skills with the ability to explain complex technical concepts to non-technical stakeholders.
  • Strong problem-solving skills and attention to detail.
Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools
Azure DatabricksPySparkSQLPostgreSQLETLELTApache AirflowAzure Kubernetes ServiceTerraformBicep
Soft Skills
communicationproblem-solvingattention to detailleadershipcollaborationtechnical expertiseongoing learningagile methodology