Intellectsoft

Senior Data Engineer, Python

Intellectsoft

full-time

Posted on:

Location Type: Hybrid

Location: Bengaluru • India

Visit company website

Explore more

✨ AI Apply
Apply

Job Level

Senior

Tech Stack

AirflowApacheAWSAzureCloudETLGoogle Cloud PlatformPySparkPythonSparkSQL

About the role

  • Design and build highly reliable and scalable data pipelines using PySpark and big data technologies.
  • Collaborate with the data science team to develop new features that enhance model accuracy and performance.
  • Create standardized data models to improve consistency across various deployments.
  • Troubleshoot and resolve issues in existing ETL pipelines and optimize workflows.
  • Conduct POCs to evaluate new technologies and integrate additional data sources.
  • Follow and promote best practices for software development, ensuring high-quality solutions that meet requirements and deadlines.
  • Document development updates and maintain clear technical documentation.

Requirements

  • 4+ years of professional experience, including 2+ years of data engineering with Apache Spark and SQL.
  • Proficiency in Python for data processing and automation.
  • Knowledge of PySpark, distributed computing, analytical databases and other big data technologies.
  • Expertise in designing and managing ETL pipelines and distributed data processing frameworks.
  • Strong knowledge of database systems, data modeling, and analytical databases.
  • Hands-on experience with workflow orchestration tools such as Apache Airflow.
  • Familiarity with cloud platforms like AWS, GCP, or Azure.
  • Solid understanding of software development lifecycles, including coding standards, version control, and testing.
  • **Nice to have skills**
  • Bachelor’s or master’s degree in Computer Science or a related field.
  • Familiarity with the data science and machine learning development process.
  • Understanding of Machine Learning pipelines or frameworks.
Benefits
  • Awesome projects with an impact
  • Udemy courses of your choice
  • Team-buildings, events, marathons & charity activities to connect and recharge
  • Workshops, trainings, expert knowledge-sharing that keep you growing
  • Clear career path
  • Absence days for work-life balance
  • Flexible hours & work setup - work from anywhere and organize your day your way
Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools
PySparkSQLPythonETL pipelinesdata modelingdistributed computinganalytical databasesworkflow orchestrationApache AirflowMachine Learning
Soft Skills
collaborationtroubleshootingproblem-solvingdocumentationbest practicescommunicationtime managementattention to detailadaptabilitycritical thinking
Certifications
Bachelor’s degree in Computer ScienceMaster’s degree in Computer Science