Weekday (YC W21)

Senior Pipeline Engineer

Weekday (YC W21)

full-time

Posted on:

Origin:  • 🇮🇳 India

Visit company website
✨ AI Apply
Manual Apply

Job Level

Senior

Tech Stack

Amazon RedshiftAWSAzureBigQueryCloudDistributed SystemsDockerGoogle Cloud PlatformJavaKafkaKubernetesPulsarPython

About the role

  • Role for one of Weekday’s clients
  • Min Experience: 6 years
  • Location: India
  • JobType: full-time
  • Design, develop, and maintain robust and scalable data pipelines for batch and real-time processing
  • Automate workflows for data ingestion, transformation, validation, and delivery
  • Ensure pipelines are fault-tolerant, performant, and cost-efficient
  • Build and manage orchestration workflows using Celery, Temporal, or similar tools
  • Monitor, schedule, and optimize pipeline execution to reduce latency and improve reliability
  • Implement scalable solutions for dependency management, retries, and error handling
  • Create efficient database schemas and models to support large-scale data operations
  • Optimize complex queries, indexing, and partitioning strategies for high performance
  • Partner with product managers, analysts, and business stakeholders to translate requirements
  • Mentor and guide junior engineers; contribute to architecture reviews and documentation
  • Proactively identify opportunities to improve pipeline performance and ensure data governance and compliance

Requirements

  • 6+ years of experience in pipeline engineering, data engineering, or related fields
  • Strong hands-on expertise in Celery, Temporal, or other orchestration frameworks
  • Proficiency in database design, query optimization, and performance tuning
  • Solid programming background in Python, Java, or similar languages
  • Deep understanding of distributed systems, messaging queues, and event-driven architectures
  • Experience with cloud platforms (AWS, Azure, GCP) and containerized environments (Docker, Kubernetes)
  • Strong problem-solving, analytical, and debugging skills
  • Excellent communication and collaboration abilities
  • Familiarity with real-time data streaming platforms like Kafka or Pulsar
  • Experience with data warehousing tools (Snowflake, BigQuery, Redshift)
  • Knowledge of CI/CD pipelines, DevOps practices, and infrastructure automation