Job description:
#### **About Us**
We’re building the first end-to-end testing platform for web agents, including a Browser Gym for RL-driven optimization. Our platform helps teams evaluate, benchmark, and improve web agents before they go live, ensuring they can handle real-world, dynamic environments.
With synthetic user simulations, automated evaluations, and large-scale benchmarking, we’re setting a new standard for web agent testing.
We’re a YC-backed team, and this is a founding engineering role—you’ll be one of the first hires defining how we crawl, structure, and analyze the open web at scale.
#### **The Role**
We need a Founding Web Scraping Engineer to build internet-scale web crawling infrastructure—not just scraping a single site, but handling millions of domains and evolving anti-bot defenses.
You’ll be responsible for designing robust, distributed crawling systems that adapt dynamically to web changes, optimize for efficiency, and ensure reliable data extraction.
#### **What You’ll Do**
* Build large-scale, distributed crawlers that intelligently prioritize, schedule, and optimize requests across millions of domains.
* Develop adaptive web scraping systems that handle DOM changes, WebSockets, AJAX-heavy sites, and dynamically loaded content.
* Optimize scraping performance and resilience, ensuring high-throughput data extraction with proxy/network optimizations and behavior-driven stealth tactics.
* Solve captchas at scale, integrating third-party solvers, heuristic-based workarounds, and behavior-driven bypass techniques.
* Manage proxy and identity rotation, implementing session-aware scraping, JA3/TLS fingerprint spoofing, and request signature control.
* Structure and clean extracted data for downstream analytics, AI training, and benchmarking applications.
#### **What We’re Looking For**
* Expert-level experience in large-scale web scraping & crawling (Selenium, Puppeteer, Playwright, Scrapy, undetected-chromedriver).
* Deep knowledge of anti-bot detection strategies (TLS fingerprinting, JA3 signatures, request header anomalies, and bot behavior tracking).
* Hands-on expertise with captcha-solving strategies, including leveraging APIs, OCR-based approaches, and behavior-driven evasion.
* Proven experience building efficient proxy management systems, including rotating IPs across residential, datacenter, and mobile networks.
* Proficiency in Python, Go, or JavaScript, with experience in high-performance, parallelized scraping frameworks.
* Understanding of HTTP/2, HTTP/3, WebSockets, GraphQL, and browser-based fingerprinting.
* Experience designing scalable, fault-tolerant scraping infrastructure that adapts to changes in real time.
#### **Bonus Points**
* Experience with search engine-scale crawling.
* Background in LLM-driven web extraction or RL-enhanced adaptive crawling.
* Contributions to open-source scraping tools or web automation projects.
#### **Why Join?**
* Founding role—you’ll define and own our web crawling infrastructure from day one.
* Work at internet scale—building a system that dynamically adapts and scales across millions of domains.
* YC-backed—we’re building something that doesn’t exist yet, and you’ll be part of the core team making it happen.