Responsible for data pipeline orchestration
Create and maintain code and repositories
Create large data sets for functional and performance tests
Create different ETL jobs
Develop and contribute to test framework
Debug test failures and triaging production issues
Validation and certification of the release candidate
Functional, System, Smoke testing on various data pipelines
Building tools for validation, debugging of data pipelines
Work with peer Data Engineers, engineers, developers and other members of the team to understand the complex systems and develop solutions for the same
Understands the business requirements and author designs and PoC’s for the Data Engineering projects
Build automation tests that fit in CI/CD pipeline
Participate in code reviews

Requirements

Strong software engineering fundamentals - Object Oriented design, data structures, and algorithms
Experience in building frameworks and solutions for Data platforms, focusing on the BI pipelines
Broad understanding of cloud architecture tools and services, such as S3, EMR, Glue, Athena, Kafka, Kubernetes, Lambda functions. Experience in AWS is highly desirable
Rich experience and deep expertise in the areas of Big Data and large scale data platforms, especially in Data Lake
Stream processing engines – Spark Structured Streaming/Kafka
Analytical processing on Big Data using Spark
Hands-on administration, configuration management, monitoring, performance tuning of Spark Batch and Streaming workloads, Distributed platforms, and JVM based systems
Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases
Excellent written and verbal communication skills
Experience in ETL testing (Billing or Data Warehouse) is a plus
Knowledge of automation, CI/CD tools like Jenkins, CloudBees, Spinnaker, others
5-8+ years of overall experience in developing / maintaining large-scale, distributed production class software on public cloud platforms such as AWS
BS degree in Computer Science or similar Engineering discipline, or equivalent work experience.

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

data pipeline orchestrationETL jobstest framework developmentdebuggingautomation testsObject Oriented designdata structuresalgorithmsSQLBig Data

communicationcollaborationproblem-solvinganalytical thinkingdesign authoring

BS degree in Computer Science