Data Engineer
Fiserv
full-time
Posted on:
Location Type: Office
Location: Nenagh • 🇮🇪 Ireland
Visit company websiteJob Level
Mid-LevelSenior
Tech Stack
AirflowAmazon RedshiftAWSCloudDockerETLHadoopJavaKafkaKubernetesLinuxMicroservicesPythonScalaSparkSQL
About the role
- Developing ETL pipelines in and out of data warehouse using combination of Java/Scala/Python Spark jobs for data transformation and aggregation
- Writing SQL queries against Snowflake
- Provide production support for Data Warehouse issues such data load problems, transformation translation problems
- Develop unit tests for transformations and aggregations
- Develop production grade real time or batch data integrations between systems
- Real time data processing of events from Kafka using streaming processing into data warehouse
- Design and build data pipelines of medium to high complexity
- Translate requirements for BI and Reporting to Database design and reporting design
- Understanding data transformation and translation requirements and which tools to leverage to get the job done
- Design and build machine learning pipelines of medium to high complexity
- Execute practices such as continuous integration and test-driven development to enable the rapid delivery of working code
- Deploy production grade data pipelines, data infrastructure and data artifacts as code
- Develop estimates for data driven solutions
- Communicate technical, product and project information to stakeholders
- Establish standards of good practice such as coding standards and data governance
- Peer review code developed by others
Requirements
- Minimum BSc or BTech / B.E in Computer Science, Engineering, or related discipline
- Relevant professional qualification such as AWS Certified Big Data, SnowPro Core certification, other Data Engineer certifications
- Strong development hands-on background in creating Snow pipe and complex data transformations and manipulations using Snow Pipe, Snow SQL
- Hands-on experience with Snowflake external tables concepts, Staging, Snow scheduler & performance tuning
- Good understanding of Snowflake Time travel concepts and zero-copy cloning, Network policies, clustering, and tasks
- 5+ years experience working in an enterprise big data environment
- Deep knowledge of Spark, Kafka and data warehouse such as snowflake, Hive, Redshift etc
- Hands-on experience in development, deployment and operation of data technologies and platforms such as Integration using APIs, micro-services and ETL patterns
- Low latency/Streaming, batch and micro batch processing
- Data platforms such as Hadoop, Hive, Redshift or Snowflake
- Cloud Services such as AWS
- Cloud query services such as Athena
- DevOps Platforms such as Gitlab
- Containerisation technologies such as Docker and Kubernetes
- Orchestration solutions such as Airflow
- Deep knowledge of key non-functional requirements such as availability, scalability, operability, and maintainability
- Deep knowledge of SQL
- OS knowledge particularly Linux
- High on social skills, team spirit and empathy
- Willingness to take ownership and ability to show strong personal commitment for the department and the team goals; must be comfortable with being considered a reliable and proactive influential team member who is not afraid to take on responsibility in the team
- Ability to communicate clearly to business analysts and stakeholders as well as technical representatives
- Very strong and proven communication and coordination skills, open-minded and determined
Benefits
- Health insurance
- Retirement plans
- Paid time off
- Flexible work arrangements
- Professional development
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard skills
ETL pipelinesJavaScalaPythonSQLSnowflakeSparkKafkamachine learning pipelinesdata transformation
Soft skills
communicationteam spiritempathyownershipcommitmentcoordinationinfluenceproactivityopen-mindednessdetermination
Certifications
AWS Certified Big DataSnowPro Core certificationData Engineer certifications