Rackspace Technology

Lead Azure Data Engineer

Rackspace Technology

full-time

Posted on:

Location Type: Hybrid

Location: Gurgaon • India

Visit company website

Explore more

✨ AI Apply
Apply

Job Level

Senior

Tech Stack

AzureCloudETLKafkaPySparkSparkSQLUnity

About the role

  • Lead end-to-end development of scalable data pipelines and orchestration frameworks using Azure Data Factory (ADF), Azure Synapse Analytics, Azure Databricks, and Microsoft Fabric.
  • Build robust real-time and batch data pipelines, including integration with streaming sources (e.g., Event Hubs, Kafka) and structured streaming engines.
  • Design and implement Structured Streaming applications in Spark for near-real-time processing of streaming data.
  • Develop and maintain ETL/ELT pipelines and transformations leveraging Spark, PySpark, SQL, and fabric orchestration capabilities.
  • Architect and implement data solutions using Microsoft Fabric, including OneLake, Dataflows, warehouses, and Fabric capacity planning to support enterprise analytics.
  • Collaborate on data governance, cataloging, and asset organization using Unity Catalog within Databricks and Fabric environments.
  • Manage Microsoft Fabric capacity and resource utilization to optimize performance and cost efficiency for analytics workloads.
  • Design, deploy, and optimize Databricks dashboards and reporting artifacts for business stakeholders.
  • Apply best practices for data modelling, caching, file sizing, and performance tuning of Spark and Delta Lake jobs (e.g., Z-ORDER, broadcast joins, adaptive query execution).
  • Oversee governance, access controls, metadata management, and lineage using Unity Catalog.
  • Lead and mentor a team of data engineers, fostering best practices in development, operations, documentation, and quality.
  • Work with cross-functional teams (architecture, BI, data science, DevOps) to translate business requirements into scalable data solutions.
  • Partner with stakeholders to define data strategy, standards, and architectural roadmaps.
  • Establish and enforce standards for data quality, testing, monitoring, operational observability, and governance.
  • Implement secure, compliant data access and lineage frameworks across cloud data platforms.
  • Implement CI/CD pipelines, infrastructure-as-code for data platform artifacts, and automated testing frameworks for data jobs and workflows.

Requirements

  • 10+ years of hands-on experience in data engineering on Azure with deep expertise in ADF, Synapse, Databricks, and Microsoft Fabric.
  • Proven experience with real-time data processing, streaming architectures, and Spark Structured Streaming.
  • Strong proficiency in Azure Data Factory, Spark (PySpark), SQL, Azure Synapse Analytics, Databricks Runtime, and cloud storage.
  • Solid knowledge of Unity Catalog for data governance, security, and access management.
  • Experience designing and managing Databricks Dashboards, performance optimization, cost controls, and data platform resource tuning.
  • Expertise in building scalable, fault-tolerant, and high-throughput batch & streaming data solutions.
  • Excellent leadership, cross-team collaboration, and communication skills.
Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools
data engineeringdata pipelinesETLELTSparkPySparkSQLStructured StreamingAzure Data FactoryMicrosoft Fabric
Soft Skills
leadershipcross-team collaborationcommunication