Relomote
True Talent Solutions Partners logo

Research Crawling Engineer

True Talent Solutions PartnersEngineering & Development
Salary
$160K–$250K
Hiring from
Probably Worldwide
Work type
Remote
Posted
Apply for this job100% free

618,681 remote jobs, straight from company career pages

100% free · New jobs every minute

Show job description

About the Role

Join a data infrastructure team building large-scale web crawling systems and datasets for AI research and model development. This role focuses on reliable data acquisition across diverse, changing web environments.

What You'll Do

  • Build and maintain web crawlers across social, travel, and multilingual websites.

  • Design high-throughput, fault-tolerant systems that collect millions to billions of URLs per day.

  • Adapt crawling approaches to anti-bot systems, rate limits, and dynamic or JavaScript-heavy sites.

  • Develop pipelines to clean, deduplicate, filter, and normalize web data at TB-to-PB scale.

  • Build and maintain datasets for research and model training in collaboration with research teams.

  • Monitor crawl performance, coverage, and data quality, and adjust systems as websites change.

  • Optimize crawling infrastructure for cost, latency, and reliability across cloud or bare-metal environments.

What We're Looking For

  • At least 2 years of experience building web crawlers or large-scale data pipelines; typically 4 or more years, or 2 or more years with a PhD.

  • Hands-on experience building and operating web crawlers at scale, including data pipelines handling TB-scale workloads.

  • Strong proficiency in Go, Rust, Python, Java, or C++, with experience in distributed systems or parallel processing.

  • Deep knowledge of HTTP, networking, and browser behavior, including experience working with anti-bot systems, rate limits, and JavaScript-heavy sites.

  • Experience with headless browsers, proxy systems, IP rotation, and request orchestration is valuable.

  • Familiarity with data quality evaluation, large-scale data processing, and cloud or bare-metal infrastructure.

  • Experience with NLP pipelines, machine learning dataset curation, or AI research data is a plus.

  • A degree in computer science, engineering, or a related technical field is helpful; a PhD is a plus.

Compensation & Benefits

Salary range: $160,000 to $250,000 annually. Visa sponsorship is not available.

Location

Remote.

Similar jobs