42dot

Senior AI Data Pipeline Engineer

42dot

full-time

Posted on:

Location Type: Hybrid

Location: Pangyo • South Korea

Visit company website

Explore more

✨ AI Apply
Apply

Job Level

Senior

Tech Stack

AirflowApacheCloudKafkaKubernetesPythonSpark

About the role

  • Design and build high-performance, scalable data pipelines to support diverse AI and Machine Learning initiatives across the organization.
  • Architect and implement multi-region data infrastructure to ensure global data availability and seamless synchronization.
  • Develop flexible pipeline architectures that allow for complex branching and logic isolation to support multiple concurrent AI projects.
  • Optimize large-scale data processing workloads using Databricks and Spark to maximize throughput and minimize processing costs.
  • Maintain and evolve the containerized data environment on Kubernetes, ensuring robust and reliable execution of data workloads.
  • Collaborate with AI researchers and platform teams to streamline the flow of high-quality data into training and evaluation pipelines.

Requirements

  • Extensive professional experience in building and operating production-grade data pipelines for massive-scale AI/ML datasets.
  • Strong proficiency in distributed processing frameworks, particularly Apache Spark and the Databricks ecosystem.
  • Deep hands-on experience with workflow orchestration tools like Apache Airflow for managing complex dependency graphs.
  • Solid understanding of Kubernetes and containerization for deploying and scaling data processing components.
  • Proficiency in distributed messaging systems such as Apache Kafka for high-throughput data ingestion and event-driven architectures.
  • Expert-level programming skills in Python for system-level optimizations.
  • Strong knowledge of cloud-native services and best practices for building secure and scalable data infrastructure.
  • Logical approach to problem-solving with the persistence to identify and resolve root causes in complex, large-scale systems.
  • Strong communication skills to effectively collaborate with cross-functional teams and external partners.
Benefits
  • 이력서 제출 시 주민등록번호, 가족관계, 혼인 여부, 연봉, 사진, 신체조건, 출신 지역 등 채용절차법상 요구 금지된 정보는 제외 부탁드립니다.
  • 모든 제출 파일은 30MB 이하의 PDF 양식으로 업로드를 부탁드립니다. (이력서 업로드 중 문제가 발생한다면 지원하시고자 하는 포지션의 URL과 함께 이력서를 recruit@42dot.ai으로 전송 부탁드립니다.)
  • 인터뷰 프로세스 종료 후 지원자의 동의하에 평판조회가 진행될 수 있습니다.
  • 국가보훈대상자 및 취업보호 대상자는 관계법령에 따라 우대합니다.
  • 장애인 고용 촉진 및 직업재활법에 따라 장애인 등록증 소지자를 우대합니다.
Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools
data pipelinesAIMachine LearningApache SparkDatabricksKubernetesApache AirflowApache KafkaPythoncloud-native services
Soft Skills
problem-solvingcommunicationcollaborationpersistence