Apply

Ready to go for it?

AI Apply speeds things up—apply directly if you prefer.

FREE ACCESS
5,000–10,000 jobs/day
JobTailor Logo

See all jobs on JobTailor

Search thousands of fresh jobs every day.

Discover
  • Fresh listings
  • Fast filters
  • No subscription required
Create a free account and start exploring right away.
Wynd Labs

Research Crawling Engineer

Wynd Labs

Research Crawling Engineer designing and operating large-scale web data acquisition systems for a bandwidth-sharing network powering AI models.

Posted 4/20/2026full-timeRemote • 🇺🇸 United StatesMid-LevelSeniorWebsite

Tech Stack

Tools & technologies
Distributed SystemsGoJavaJavaScriptPythonRust

About the role

Key responsibilities & impact
  • Build and maintain large-scale web crawlers across diverse domains
  • Design high-throughput, fault-tolerant systems for data collection (millions to billions of URLs/day)
  • Handle anti-bot systems, rate limits, and dynamic/JS-heavy sites
  • Develop pipelines for cleaning, deduplication, filtering, and normalization
  • Construct and maintain datasets for research and model training
  • Monitor crawl performance, coverage, and data quality; iterate quickly
  • Collaborate with research teams to align data collection with modeling needs
  • Optimize infrastructure for cost, latency, and reliability

Requirements

What you’ll need
  • Strong programming experience in one or more of: Go, Rust, Python, Java, or C++
  • Experience building web crawlers or large-scale data pipelines
  • Solid understanding of HTTP, networking, and browser behavior
  • Familiarity with distributed systems and parallel processing
  • Experience working with large datasets (TB–PB scale preferred)
  • Ability to debug unstable or adversarial environments

Benefits

Comp & perks
  • Competitive salary
  • Benefits and equity package

ATS Keywords

✓ Tailor your resume
Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools
GoRustPythonJavaC++web crawlersdata pipelinesHTTPdistributed systemsparallel processing
Soft Skills
collaborationproblem-solvingdebuggingiteration