Design and implement scalable data lake solutions using Snowflake and Databricks
Develop and optimize data pipelines for ingestion, transformation, and storage
Manage data governance, quality, and security across cloud environments and implement performance tuning, automation, and CI/CD for data workflows
Collaborate with cross-functional teams to support cloud migration activities
Tune Hadoop, Hive, and Spark jobs and configurations for optimal performance, efficiency, and resource utilization
Diagnose and resolve issues related to Linux servers, networks, cluster health, job failures, and performance bottlenecks
Provide on-call support and collaborate with other teams to ensure smooth operations
Implement and manage security measures within the Cloudera environment, including Kerberos, Apache Ranger, and Atlas, to ensure data governance and compliance
Setup and manage HashiCorp Vault for secure keys and secrets management
Migrate Datastage ETL jobs to Azure cloud services such as Azure Synapse Analytics, Azure Databricks, or Snowflake
Develop scripts (e.g., shell, Ansible, Python) for automating administrative tasks, deployments, and monitoring
Create and maintain documentation for system configurations, operational procedures, and troubleshooting knowledge bases
Work closely with the vendor to stay current with the latest releases, perform upgrades, and address vulnerabilities

Requirements

Bachelor’s degree, or equivalent work experience
Three to five years of relevant experience
Deep expertise in Data Engineering and Management technologies, synthetic data, automation, advanced analytics
Ability to do on-call rotation once a month
Very strong customer-centric focus
6 - 8 years of hands-on experience in Data engineering, Cloud platform management, and performance optimization
Very strong Azure Data Factory tools experience
Excellent SQL Experience, including performance tuning and optimization
Hands-on experience with Hadoop, Hive, Spark, and migration of Big Data into Azure cloud services
DataStage experience for conversion of ETL jobs to Pyspark ETL pipelines
Working with offshore teams
Working knowledge and hands-on experience in Data Integration and Data Lake Architectures with Databricks and Snowflake platforms
Working knowledge of Microsoft Azure cloud and big data migration to cloud platforms
Proficiency in Linux, clustering, and distributed systems
Expertise in Hive and Spark for data processing and analytics
Expertise in Hadoop ecosystem components such as HDFS, YARN, Hive, Spark, and Sqoop
Proficiency in languages such as shell, Python, Pyspark for automating workflows, deployments, and monitoring
Expertise in Linux, Network, Python scripting, DNS, Kerberos, LDAP/AD, JupyterHub
Experience in creating and maintaining documentation for system configurations, operational procedures, and troubleshooting knowledge bases
Strong problem-solving skills and the ability to diagnose and resolve system failures and performance bottlenecks
Excellent communication and collaboration skills to work effectively with cross-functional teams

Benefits

Healthcare (medical, dental, vision)
Basic term and optional term life insurance
Short-term and long-term disability
Pregnancy disability and parental leave
401(k) and employer-funded retirement plan
Paid vacation (from two to five weeks depending on salary grade and tenure)
Up to 11 paid holiday opportunities
Adoption assistance
Sick and Safe Leave accruals of one hour for every 30 worked, up to 80 hours per calendar year unless otherwise provided by law

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

data engineeringdata managementdata pipelinesperformance tuningautomationSQLHadoopHiveSparkPython

Soft Skills

customer-centric focusproblem-solvingcommunicationcollaboration