Design and build data ingestion pipelines integrating with dozens of external sources (social platforms, third-party APIs, web scrapers)
Architect ETL workflows using Airflow and Kafka to process high-volume data streams (millions of records daily)
Collaborate with AI/ML engineers to ensure data quality and availability for Agentic RAG pipelines
Build and optimize multi-store data architecture: vector databases (Pinecone), relational databases (Postgres), and search engines (OpenSearch)
Develop and maintain integrations with social media platforms (Meta, LinkedIn, TikTok, X/Twitter) handling webhook ingestion and API polling
Optimize data freshness, throughput, and reliability across distributed systems
Participate in code reviews and contribute to backend services (FastAPI/Python, occasional Ruby on Rails)

Requirements

Bachelor's degree in Computer Science or related degree; or equivalent work experience
Minimum 5 years of professional software engineering experience
Minimum 4 years proven experience building data pipelines and ETL workflows in Python
Minimum 3 years experience with workflow orchestration tools (Airflow, Dagster, Prefect, or similar)
Minimum 3 years working with multiple data storage technologies (relational, vector, search engines)
Experience with message queues and event streaming (Kafka, RabbitMQ, SQS/SNS)
Proven experience building high-throughput, fault-tolerant systems (we process 100s of millions of comments per year)
Strong understanding of API design, rate limiting, and webhook handling
Experience with data quality monitoring and observability
Demonstrated ability to take ownership of projects, prioritize tasks, and deliver high-quality results independently

Benefits

📊 Check your resume score for this job Improve your chances of getting an interview by checking your resume score before you apply. Check Resume Score

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

data ingestionETL workflowsAirflowKafkaPythonFastAPIRuby on Railsdata pipelinesAPI designdata quality monitoring

Soft Skills

collaborationproject ownershiptask prioritizationindependent deliverycommunication

Certifications

Bachelor's degree in Computer Science