Research Crawling Engineer

About the Role

Join a data infrastructure team building large-scale web crawling systems that supply public web data for AI research and model development. You will work closely with research teams to shape data collection around their modeling needs.

What You'll Do

• Build and maintain web crawlers across diverse domains, including social, travel, and multilingual websites.

• Design high-throughput, fault-tolerant systems that collect millions to billions of URLs per day.

• Adapt collection methods to anti-bot protections, rate limits, and dynamic, JavaScript-heavy sites.

• Build pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.

• Develop and maintain datasets for research and model training in collaboration with research teams.

• Monitor crawl coverage, performance, and data quality, and respond to changing web environments.

• Optimize crawling infrastructure for cost, latency, and reliability across cloud and bare-metal environments.

What We're Looking For

• At least 2 years of experience building web crawlers or large-scale data pipelines; typically 4 or more years, with 2 or more years considered for candidates with a PhD.

• Experience operating data pipelines at terabyte scale and building distributed or parallel systems.

• Strong programming skills in Go, Rust, Python, Java, or C++.

• Deep knowledge of HTTP, networking, and browser behavior, with experience handling anti-bot systems, rate limits, or JavaScript-heavy sites.

• Experience with headless browsers such as Playwright or Puppeteer, proxy systems, and IP rotation.

• Background in data quality evaluation, web data processing, NLP pipelines, or ML dataset curation is beneficial.

• Experience running data workloads on cloud platforms or bare-metal infrastructure.

Compensation & Benefits

Base salary: $160,000 to $250,000 USD annually. Visa sponsorship is not available.

Location

Remote.