Software Mind

Senior Data Engineer – Azure

Software Mind

full-time

Posted on:

Location Type: Remote

Location: Poland

Visit company website

Explore more

✨ AI Apply
Apply

Job Level

Senior

Tech Stack

CloudETLKafkaPySparkPythonSparkSQL

About the role

  • Develop reusable, metadata-driven data pipelines
  • Automate and optimize any data platform related processes
  • Build integrations with data sources and data consumers
  • Add data transformation methods to shared ETL libraries
  • Write unit tests
  • Develop solutions for the Databricks data platform monitoring
  • Proactively resolve any performance or quality issues in ETL processes
  • Cooperate with infrastructure engineering team to set up cloud resources
  • Contribute to data platform wiki / documentation
  • Perform code reviews and ensures code quality
  • Initiate and implements improvements to the data platform architecture

Requirements

  • Programming: Python/PySpark, SQL
  • Proficient in building robust data pipelines using Databricks Spark
  • Experienced in dealing with large and complex datasets
  • Knowledgeable about building data transformations modules organized as libraries (Python packages)
  • Familiar with Databricks Delta optimization techniques (partitioning, z-ordering, compaction, etc.)
  • Experienced in developing CI/CD pipelines
  • Experienced in leveraging event brokers (Kafka /Event Hubs / Kinesis) to integrate with data sources and data consumers
  • Understanding of basic networking concepts
  • Familiar with Agile Software Development methodologies (Scrum)
Benefits
  • Flexible employment and remote work
  • International projects with leading global clients
  • International business trips
  • Non-corporate atmosphere
  • Language classes
  • Internal & external training
  • Private healthcare and insurance
  • Multisport card
  • Well-being initiatives
Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools
PythonPySparkSQLDatabricksETLCI/CDKafkaEvent HubsKinesisAgile
Soft Skills
code qualityproblem solvingcollaborationdocumentationperformance optimization