Data Platform Manager

A.C.Coy Company

Data Platform Manager at A.C.Coy, leading a team to design and optimize data platforms. Focus on Azure Databricks and cloud technologies in a hybrid work environment.

Posted 6/4/2026full-timePittsburgh • Pennsylvania • 🇺🇸 United StatesMid-LevelSeniorWebsite

Tech Stack

Tools & technologies

AirflowApacheAzureCloudETLHadoopKafkaNoSQLPySparkPythonScalaSparkSQLUnityVault

About the role

Key responsibilities & impact

Lead and mentor a team of data engineers, conducting code reviews and ensuring development standards
Support troubleshooting and incident management for data-related issues in production
Collaborate with business stakeholders, data scientists, and other team members to gather requirements and translate them into technical specifications
Lead the design, development and deployment of scalable and high-performance data pipelines using Azure Databricks
Optimize pipeline performance, cost, and scalability in the Azure cloud environment
Implement data quality checks and validation procedures to ensure the accuracy and integrity of data
Collaborate with data scientists and analysts to operationalize and deploy machine learning models
Define the end-to-end Lakehouse architecture using Delta Lake, implementing medallion architecture (Bronze, Silver, Gold layers)
Oversee the development of robust, scalable batch and streaming ETL/ELT pipelines using PySpark, Scala, and SQL
Integrate real-time and batch data sources using Apache Kafka and ADF
Implement Unity Catalog for unified governance, data security, fine-grained access control (RBAC), privacy measures, and data lineage tracking
Tune Spark jobs and Databricks clusters to maximize throughput while maintaining cost efficiency

Requirements

What you’ll need

Bachelor's degree in Computer Science, Engineering, or a related field
5-7+ years hands-on data engineering or architecture, with at least 2-4 years specifically focused on Azure Databricks, including Azure cloud technologies
2-5 years experience in managing a team of data engineers, data scientists and/or analysts
Certifications (Preferred): Microsoft Certified: Azure Data Engineer Associate (DP-203), Databricks Certified Data Engineer Professional, or Azure Solutions Architect Expert
Proficiency in both Relational (SQL) and NoSQL (Document, Key-Value, Graph, Columnar) databases
Knowledge of frameworks like Apache Hadoop, Spark, or Presto/Trino
Understanding file formats like Parquet, Avro, or ORC and compression techniques
Deep proficiency in programming languages: Python (specifically PySpark), SQL, PowerShell, and Scala
Hands-on experience with Azure Cloud infrastructure, including Networking (VNETs), Key Vault, and Identity Management
Deep knowledge of Apache Spark runtime internals, MLflow for MLOps, and orchestration tools like Airflow

Benefits

Comp & perks

No C2C or Sponsorship

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

data engineeringdata architectureAzure DatabricksPySparkScalaSQLETLELTmachine learningdata quality checks

Soft Skills

leadershipmentoringcollaborationtroubleshootingincident managementcommunicationrequirement gatheringtechnical specification translationteam managementperformance optimization

Certifications

Microsoft Certified: Azure Data Engineer Associate (DP-203)Databricks Certified Data Engineer ProfessionalAzure Solutions Architect Expert