Scrape sites that fight back
z3l.Scraper is a crawling engine for .NET. You write what to take from a page; it deals with 403s, captchas, 429s, cut-off pages, flaky servers and crashed runs.
> dotnet add package z3l.Scraper --prerelease
Your code is only what to take
Queues, retries, checks, limits and state are the engine's job.
- 1Labels and typed page dataEvery kind of page has its handler, by
PageTypeor your own label; a page passes what it knows to the next one, typed. - 2Checks before your codeBlocked, missing and cut-off pages never reach the handler: they are retried or reported.
- 3Whole pages or nothingItems of a handler that throws are dropped, so a retried page never gives duplicates.
- 4Results as you go
await foreachgets items while the crawl runs; a slow consumer slows the crawl down.
var proxies = ProxyList.FromFile("proxies.txt"); // http, socks5...
using var loader = new HttpPageLoader(new() { Proxies = proxies });
var scraper = new Scraper(loader)
.Route(PageType.Listing, ctx =>
{
foreach (var card in ctx.Html.FindAll(HtmlTag.Article))
ctx.Enqueue(card.Find(HtmlTag.A).AttributeHref(), PageType.Item,
new Teaser(card.Find(HtmlTag.P, cls: "rating").Text()));
ctx.Enqueue(ctx.Html.XPathValue("//li[@class='next']/a/@href"), PageType.Listing);
})
.Route<Teaser>(PageType.Item, (ctx, teaser) => ctx.Emit(new Book(
ctx.Html.Find(HtmlTag.H1).Text(), teaser.Rating)));
await foreach (var item in scraper.RunAsync(
new ScrapeRequest("https://books.toscrape.com/", PageType.Listing)))
{
if (item is Book book) Save(book); // streamed while the crawl runs
}
When the site fights back
Every case below is covered by the test suite; most run against a local shop that misbehaves on purpose.
| The site does | z3l.Scraper |
|---|---|
| 403, 429, a captcha, "Just a moment..." | Recognised as a block: the proxy that got it rests for that site (an hour by default), the page is retried through another proxy. With AutoThrottle (for scraping from one address) the host slows down too. |
| A proxy dies, hangs or wants a login | One failure puts it after the clean proxies; three in a row and it rests for every site. |
| A proxy answers with its own page | The page fails MustContain and is retried; the proxy goes after the clean ones for that site, three such answers in a row and it rests there. |
| 429 with Retry-After | The page is retried as late as asked. With AutoThrottle the whole host pauses; pages of other hosts keep loading. |
| 500, 503, a timeout, a dropped connection | Retried with exponential backoff and jitter, up to the limit you set. |
| A page cut off half-way, an empty body, broken JSON | Retried: ExpectedContent.Html, Json, MustContain per kind of page. |
| 404 or "Product not found" | Reported as not found after one more try, not retried forever. |
| Prices hidden for bots: looks fine, isn't | Your handler throws RetryRequestException: retried and counted as a block. |
| robots.txt rules and Crawl-delay | Disallowed pages are skipped and loads are spaced, with RespectRobotsTxt. |
| Your process dies at page 40,000 | The next run continues from the state file. Pages whose items you had not taken are done again: nothing is lost. |
Everything a crawl needs
In one package of about 3,300 lines, with tests for every case.
Queue with priorities
De-duplicated by method, URL and body. Link depth, MaxDepth, MaxRequests, MaxRequestsPerMinute, time limits.
Response checks
Blocked, not found, incomplete: status codes, markers and your own checks, per label, before your handler runs.
Polite retries
Backoff with jitter, Retry-After, and a per-host throttle that slows a struggling site down and speeds up again.
Sitemaps and robots.txt
Sitemap indexes, gzip and text sitemaps in one call; robots.txt matched by RFC 9309 rules, with Crawl-delay.
Runs that survive a crash
Set StatePath: the queue is saved as it goes and the next run continues. At least once, never lost.
Proxies, or any loader
HttpClient with rotating http and socks proxies from a text file built in; plug a browser or anything else with IPageLoader. Trimming and AOT compatible.
What it is not. It does not run JavaScript: for pages built in the browser, plug a browser loader
(Playwright, Selenium, a remote farm) into IPageLoader. The queue lives in one process and its state file,
not in a distributed store. Proxies come from your list: it does not buy or find them for you.
Simple, one-time pricing
Free for individuals and small companies. Larger organizations pay once and use it forever.
Community
Individuals and organizations under USD 1M annual revenue and under 100 employees.
- Commercial use included
- All features, no limits
- All future versions
- Community support on GitHub
No registration, no license key.
Commercial
Organizations with USD 1M or more annual revenue, or 100 or more employees, including affiliates.
- Covers all z3l libraries, z3l.FastHtmlParser included
- Perpetual license, all current and future versions
- Unlimited developers, apps, servers and deployments
- Invoice and license certificate for your records
- Priority email support
Nothing else to buy: one license for every z3l library.
Questions
Do I need a commercial license?
Only if your organization, together with its affiliates, has USD 1,000,000 or more in annual revenue, or 100 or more employees and contractors. Everyone else uses it free, including in commercial products.
Do I need a license for z3l.FastHtmlParser too?
No. The scraper parses pages with z3l.FastHtmlParser; it is free under the thresholds, and above them the same USD 1,000 commercial license covers it, along with every other z3l library. You pay once.
Does it run JavaScript?
No. The built-in loader is HttpClient. For pages built by scripts, implement IPageLoader with Playwright, Selenium or a remote browser farm; the queue, checks and retries stay the same.
What happens if my process crashes?
With StatePath set, the run state is saved every 30 seconds and when the run stops. The next run continues: pages being loaded and items you had not taken yet are done again, nothing is lost.
Does it obey robots.txt?
When you turn on RespectRobotsTxt: robots.txt of each host is read once, disallowed pages are skipped and its Crawl-delay spaces the loads. Scraping within the law and the sites' terms is up to you.
Does the library phone home or check a license key?
No. There is no license key, no telemetry and no network access other than the pages you load.
Point it at a site
Install the package and run the quick start against books.toscrape.com, a sandbox made for scraping practice.
> dotnet add package z3l.Scraper --prerelease