Datavail

Databricks Consultant

Datavail

full-time

Posted on:

Location Type: Hybrid

Location: Mumbai • India

Visit company website

Explore more

✨ AI Apply
Apply

Tech Stack

ApacheAWSAzureCloudETLGoogle Cloud PlatformPySparkPythonRDBMSSparkSQLUnity

About the role

  • Build scalable ETL/ELT pipelines using Databricks (PySpark, SQL, Spark Streaming).
  • Develop and optimize Delta Lake tables, ACID transactions, schema evolution, and time travel.
  • Implement Unity Catalog, data governance, and access control.
  • Optimize cluster configurations, job workflows, and performance tuning in Databricks.
  • Design and implement batch and streaming pipelines using Spark Structured Streaming.
  • Integrate Databricks with multiple data sources (RDBMS, APIs, cloud storage, message queues).
  • Develop reusable, modular, and automated data processing frameworks.
  • Implement CI/CD pipelines for Databricks using GitHub Actions / Azure DevOps / GitLab.
  • Automate cluster management and job orchestration using Databricks REST APIs.
  • Maintain code quality, unit tests, and documentation.
  • Write and optimize complex SQL queries and statements to ensure high performance and efficient data retrieval.
  • Conduct performance analysis, troubleshoot database issues like slow queries or deadlocks and implement solutions.
  • Design and implement database structures, including tables, schemas, views, stored procedures, functions, and triggers.
  • Optimize database performance through query tuning, indexing, and performance analysis.
  • Ensure data integrity, security, and compliance standards.
  • Need strong Python skills combined with expertise in Apache Spark for large scale data processing.
  • Collaborate with data engineers and scientists to implement workflows, conduct code reviews, and integrate with cloud platforms like AWS or Azure.

Requirements

  • 5+ years of experience
  • Strong hands-on experience with Databricks, PySpark, and Spark SQL.
  • Expertise in Delta Lake, Bronze–Silver–Gold architecture, and Lakehouse patterns.
  • Strong experience with cloud platforms (AWS/Azure/GCP).
  • Solid understanding of data warehousing, dimensional modeling, and big-data concepts.
Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools
ETLELTDatabricksPySparkSQLSpark StreamingDelta LakeCI/CDdata processing frameworksdatabase performance tuning
Soft Skills
collaborationtroubleshootingcode reviewsdocumentationperformance analysis