Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of sources like Hadoop, Spark, AWS Lambda, etc.
Experience with AWS Cloud on Data Integration with Apache Spark, EMR, Glue, Kafka, Kinesis and Lambda in S3, Redshift, RDS, and MongoDB/DynamoDB ecosystems.
Strong real-life experience in Python development, especially in PySpark in AWS Cloud environment.
Design, develop, test, deploy, maintain and improve data integration pipeline.
Develop pipeline objects using Apache Spark / Pyspark / Python or Scala.
Design and develop data pipeline architectures using Hadoop, Spark and related AWS Services.
Load and performance test data pipelines built using the above-mentioned technologies.

Requirements

Bachelor or Master's degree in Computer Science, or related field is required.
8+ years of hands-on professional work experience with AWS and Python programming, experience with Python frameworks is required
Hands-on expertise with cloud platforms including AWS and GCP
Expert level knowledge of using SQL to write complex, highly-optimized queries across large volumes of data.
Working experience on ETL pipeline implementation using AWS services such as Glue, Lambda, EMR, Shell, S3, SNS, Pyspark, etc. is required.
Strong knowledge of data warehousing solutions, particularly Amazon Redshift
Hands-on professional work experience using emerging technologies (Snowflake, Talend, and/or Databricks) is highly desirable.
Proficiency in DBT for data transformation and modeling
Experience with automation of data workflows and processes
Strong problem solving and troubleshooting skills with the ability to exercise mature judgement.

Benefits

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

PythonPySparkScalaSQLETLData IntegrationData Pipeline ArchitectureData WarehousingDBTAutomation

Soft Skills

Problem SolvingTroubleshootingJudgement

Certifications

Bachelor's degreeMaster's degree