Trying to fix the web scraping industry's benchmark problem

Details

External ID

48935383

Source

HN

Company

—

Product

Trying to fix the web scraping industry's benchmark problem

Website domain

github.com

Launched

July 16, 2026

Cohort

—

Upvotes

18

Upvotes percentile

0.6893667861409797

Tags

—

Fetched at

Sept. 7, 2026, 9:26 p.m.

Updated at

Sept. 7, 2026, 9:26 p.m.

Description

We've been trying to evaluate web scraping companies, but when you look at their benchmarks, you can't verify anything, and they mostly exist to prove the company is successful. They put somewhere between 98% and 100% because they pick their own urls, define success their own way, and don't publish the harness. We also saw companies like scrapfly astroturf websites like scrapeway and call them independent.So, we built an open source benchmark that we want to represent the frontier of web data. We're trying to look across all major anti-bot providers and industries, to build a comprehensive hard-target test set. It's all in the repo, you can run it, and we pledge to maintain it monthly.Curious what people think we could do to improve it? Would you change any of the conditions, add sites, etc. Our target list and pass criteria are something we're trying to improve, so would love to get feedback from the community on how we can actually provide something useful for people to test against!

• Source listing
• Website