Principal Data Engineer

Medical Guardian

Principal Data Engineer building and leading data engineering foundation for Medical Guardian. Focusing on real-time decisioning, operational applications, analytics, and data services.

Posted 7/6/2026full-timePhiladelphia • Pennsylvania • 🇺🇸 United StatesLeadWebsite

Tech Stack

Tools & technologies

AzureCloudDistributed SystemsETLIoTMicroservicesPySparkPythonSparkSQL

About the role

Key responsibilities & impact

Design, build, optimize, and operate production-grade batch and streaming data pipelines on Azure and Databricks, with a primary focus on real-time IoT and telemetry use cases within a Medallion architecture.
Develop ETL/ELT workflows to ingest, transform, validate, and serve large volumes of structured, semi-structured, unstructured, and streaming data.
Build and maintain reliable data products, data services, APIs, and microservices that support operational applications, analytics, software engineering, and ML/AI teams.
Use Python, PySpark, Spark SQL, SQL, Delta Lake, Databricks Workflows, CI/CD, and related tools to build maintainable, testable, and observable data systems.
Troubleshoot complex production pipeline issues across Databricks, Azure, streaming systems, APIs, and source systems, including root cause analysis, corrective action, and prevention planning.
Move quickly from rough business need to prototype, pilot, and production-ready data capability while maintaining appropriate engineering discipline.
Lead the design and delivery of real-time streaming ingestion and processing patterns for connected medical device telemetry, event data, and operational data feeds.
Implement streaming solutions using Azure Event Hubs, Azure Stream Analytics, Databricks, Delta Lake, and related Azure integration patterns.
Define reliability, latency, quality, observability, and supportability expectations for production streaming systems.
Manage, mentor, and develop data engineers, providing clear expectations, technical guidance, prioritization support, feedback, and accountability.

Requirements

What you’ll need

10+ years of professional experience in data engineering, software engineering, data platform engineering, distributed systems, analytics engineering, or related technical fields.
7+ years of hands-on experience designing, building, optimizing, and operating production data pipelines, data platforms, or data services.
5+ years of hands-on experience with modern cloud data platforms, including Databricks, Spark, Delta Lake, SQL, Python/PySpark, and production pipeline orchestration.
3+ years of experience leading, managing, mentoring, or providing technical direction to data engineers or related technical teams.
Strong experience with Azure cloud services for data engineering, streaming, integration, storage, security, and production operations.
Experience designing and operating real-time streaming, event-driven, or near-real-time data pipelines in production or business-critical environments.
Experience applying DevOps, CI/CD, testing, version control, code review, documentation, and automation practices to data engineering workloads.
Experience building data services, APIs, microservices, or reusable consumption patterns for downstream applications, analytics, ML/AI, or operational workflows.
Strong understanding of data quality, observability, monitoring, lineage, reliability, cost optimization, privacy, and production support for data systems.
Experience translating ambiguous business needs into technical designs, architecture recommendations, delivery plans, and measurable outcomes.
Ability to explain data architecture, pipeline behavior, tradeoffs, assumptions, risks, and limitations to both technical and non-technical stakeholders.
Strong ownership mindset and ability to drive work forward independently in a fast-moving, evolving environment.

Benefits

Comp & perks

Health Care Plan (Medical, Dental & Vision)
Paid Time Off (Vacation, Sick Time Off & Holidays)
Company Paid Short Term Disability and Life Insurance
Retirement Plan (401k) with Company Match

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

PythonPySparkSpark SQLSQLETL/ELT WorkflowsData QualityObservabilityCI/CDProduction OperationsData Architecture

Soft Skills

Ownership MindsetTechnical CommunicationProblem-SolvingMentoringCollaboration