PortPro

Director of Software Engineering – Node.js, Web Scraping Expert

PortPro

full-time

Posted on:

Location Type: Remote

Location: California • United States

Visit company website

Explore more

✨ AI Apply
Apply

Job Level

Lead

Tech Stack

AirflowAWSAzureCloudDockerElasticSearchGoogle Cloud PlatformJavaScriptKafkaKubernetesMicroservicesMongoDBNode.jsNoSQLPostgresPuppeteerRabbitMQRedisSeleniumSparkSQL

About the role

  • Architect, develop, and maintain scalable and distributed web scraping systems using Node.js.
  • Design and implement data extraction pipelines to process large volumes of structured and unstructured data.
  • Develop solutions to bypass anti-bot mechanisms, including CAPTCHA handling, session management, fingerprinting, and IP rotation.
  • Optimize scraping processes for performance, reliability, and efficiency while managing proxy services (residential, datacenter, rotating).
  • Oversee data storage and processing strategies, ensuring high availability and consistency.
  • Collaborate with Product, DevOps, and Data Science teams to integrate extracted data into analytics and business applications.
  • Implement best practices for microservices, API integrations, and real-time data streaming.
  • Lead the transition to cloud-native, containerized, and serverless architectures for web scraping.
  • Ensure compliance with legal and ethical standards (robots.txt, GDPR, CCPA, etc.).
  • Optimize cloud resources (AWS, GCP, or Azure) to support high-throughput scraping.
  • Manage real-time monitoring and alerting systems to detect scraping failures, IP bans, or performance bottlenecks.
  • Work closely with DevOps teams to optimize CI/CD pipelines, automated deployments, and system scalability.
  • Lead, mentor, and grow a high-performance engineering team.
  • Define and execute the technology roadmap, aligning with business objectives.
  • Foster a culture of continuous learning, collaboration, and innovation.
  • Implement agile development methodologies (Scrum, Kanban) to optimize project execution.
  • Ensure code quality, security, and best practices across all engineering efforts.

Requirements

  • 10+ years of experience in software engineering, with at least 5+ years in web scraping and large-scale data extraction.
  • Strong hands-on expertise in Node.js, Puppeteer, Playwright, Cheerio, Selenium, and headless browser automation.
  • Extensive experience in handling CAPTCHAs, IP rotation, session management, and anti-bot evasion techniques.
  • Deep knowledge of proxy management (residential, datacenter, rotating, and VPNs).
  • Experience with NoSQL/SQL databases (MongoDB, PostgreSQL, Redis, Elasticsearch, etc.).
  • Familiarity with data processing frameworks (Kafka, RabbitMQ, Spark, Airflow, etc.).
  • Strong experience with CI/CD, containerization (Docker, Kubernetes), and cloud deployment (AWS/GCP/Azure).
Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools
Node.jsPuppeteerPlaywrightCheerioSeleniumCAPTCHA handlingsession managementIP rotationNoSQL databasesSQL databases
Soft Skills
leadershipmentoringcollaborationcontinuous learninginnovationproject executioncommunicationteam managementproblem-solvingagile methodologies