Lead Data Engineer, PySpark
Tide
ApacheETLPySparkPythonSparkSQL
AWS Data Pipeline Engineer
Kyndryl
full-time
Posted on:
Location Type: Hybrid
Location: Hyderabad • 🇮🇳 India
Visit company websiteJob Level
SeniorLead
Tech Stack
Amazon RedshiftAWSETLJavaPySparkPythonRDBMSSQLTerraform
About the role
- Design and develop scalable, secure, and fault-tolerant data pipelines utilizing AWS services such as Glue, Lambda, Kinesis, S3, EMR, Step Functions, and Athena.
- Create and maintain ETL/ELT workflows to support both structured and unstructured data ingestion from various sources, including RDBMS, APIs, SFTP, and Streaming.
- Optimize data pipelines for performance, scalability, and cost-efficiency.
- Develop and manage data models, data lakes, and data warehouses on AWS platforms (e.g., Redshift, Lake Formation).
- Collaborate with DevOps teams to implement CI/CD and infrastructure as code (IaC) for data pipelines using CloudFormation or Terraform.
- Ensure data quality, validation, lineage, and governance through tools such as AWS Glue Data Catalog and AWS Lake Formation.
- Work in concert with data scientists, analysts, and application teams to deliver data-driven solutions.
- Monitor, troubleshoot, and resolve issues in production pipelines.
- Stay abreast of AWS advancements and recommend improvements where applicable.
Requirements
- Bachelor’s or master’s degree in computer science, Engineering, or a related field
- Over 8 years of experience in data engineering
- More than 3 years of experience with the AWS data ecosystem
- Strong experience with Java, Pyspark, SQL, and Python
- Proficiency in AWS services: Glue, S3, Redshift, EMR, Lambda, Kinesis, CloudWatch, Athena, Step Functions
- Familiarity with data modelling concepts, dimensional models, and data lake architectures
- Experience with CI/CD, GitHub Actions, CloudFormation/Terraform
- Understanding of data governance, privacy, and security best practices
- Strong problem-solving and communication skills
Benefits
- Kyndryl cares about your well-being and prides itself on offering benefits that give you choice, reflect the diversity of our employees and support you and your family through the moments that matter – wherever you are in your life journey.
- Our employee learning programs give you access to the best learning in the industry to receive certifications, including Microsoft, Google, Amazon, Skillsoft, and many more.
- Through our company-wide volunteering and giving platform, you can donate, start fundraisers, volunteer, and search over 2 million non-profit organizations.
- At Kyndryl, we invest heavily in you, we want you to succeed so that together, we will all succeed.
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard skills
data engineeringETLELTdata modelingJavaPysparkSQLPythonAWS servicesCI/CD
Soft skills
problem-solvingcommunication