.NET 9 & .NET 10 · Retries · Blocks · Throttling · robots.txt · Sitemaps · Resume

Scrape sites that fight back

z3l.Scraper is a crawling engine for .NET. You write what to take from a page; it deals with 403s, captchas, 429s, cut-off pages, flaky servers and crashed runs.

> dotnet add package z3l.Scraper --prerelease
27kpages/s engine speed, in-memory loader
9kinds of site trouble handled
0pages lost when a run resumes
2dependencies

Your code is only what to take

Queues, retries, checks, limits and state are the engine's job.

  • 1
    Labels and typed page dataEvery kind of page has its handler, by PageType or your own label; a page passes what it knows to the next one, typed.
  • 2
    Checks before your codeBlocked, missing and cut-off pages never reach the handler: they are retried or reported.
  • 3
    Whole pages or nothingItems of a handler that throws are dropped, so a retried page never gives duplicates.
  • 4
    Results as you goawait foreach gets items while the crawl runs; a slow consumer slows the crawl down.
var proxies = ProxyList.FromFile("proxies.txt");   // http, socks5...
using var loader = new HttpPageLoader(new() { Proxies = proxies });

var scraper = new Scraper(loader)
    .Route(PageType.Listing, ctx =>
    {
        foreach (var card in ctx.Html.FindAll(HtmlTag.Article))
            ctx.Enqueue(card.Find(HtmlTag.A).AttributeHref(), PageType.Item,
                new Teaser(card.Find(HtmlTag.P, cls: "rating").Text()));
        ctx.Enqueue(ctx.Html.XPathValue("//li[@class='next']/a/@href"), PageType.Listing);
    })
    .Route<Teaser>(PageType.Item, (ctx, teaser) => ctx.Emit(new Book(
        ctx.Html.Find(HtmlTag.H1).Text(), teaser.Rating)));

await foreach (var item in scraper.RunAsync(
    new ScrapeRequest("https://books.toscrape.com/", PageType.Listing)))
{
    if (item is Book book) Save(book);   // streamed while the crawl runs
}

When the site fights back

Every case below is covered by the test suite; most run against a local shop that misbehaves on purpose.

The site doesz3l.Scraper
403, 429, a captcha, "Just a moment..."Recognised as a block: the proxy that got it rests for that site (an hour by default), the page is retried through another proxy. With AutoThrottle (for scraping from one address) the host slows down too.
A proxy dies, hangs or wants a loginOne failure puts it after the clean proxies; three in a row and it rests for every site.
A proxy answers with its own pageThe page fails MustContain and is retried; the proxy goes after the clean ones for that site, three such answers in a row and it rests there.
429 with Retry-AfterThe page is retried as late as asked. With AutoThrottle the whole host pauses; pages of other hosts keep loading.
500, 503, a timeout, a dropped connectionRetried with exponential backoff and jitter, up to the limit you set.
A page cut off half-way, an empty body, broken JSONRetried: ExpectedContent.Html, Json, MustContain per kind of page.
404 or "Product not found"Reported as not found after one more try, not retried forever.
Prices hidden for bots: looks fine, isn'tYour handler throws RetryRequestException: retried and counted as a block.
robots.txt rules and Crawl-delayDisallowed pages are skipped and loads are spaced, with RespectRobotsTxt.
Your process dies at page 40,000The next run continues from the state file. Pages whose items you had not taken are done again: nothing is lost.

Everything a crawl needs

In one package of about 3,300 lines, with tests for every case.

Queue with priorities

De-duplicated by method, URL and body. Link depth, MaxDepth, MaxRequests, MaxRequestsPerMinute, time limits.

Response checks

Blocked, not found, incomplete: status codes, markers and your own checks, per label, before your handler runs.

Polite retries

Backoff with jitter, Retry-After, and a per-host throttle that slows a struggling site down and speeds up again.

Sitemaps and robots.txt

Sitemap indexes, gzip and text sitemaps in one call; robots.txt matched by RFC 9309 rules, with Crawl-delay.

Runs that survive a crash

Set StatePath: the queue is saved as it goes and the next run continues. At least once, never lost.

Proxies, or any loader

HttpClient with rotating http and socks proxies from a text file built in; plug a browser or anything else with IPageLoader. Trimming and AOT compatible.

What it is not. It does not run JavaScript: for pages built in the browser, plug a browser loader (Playwright, Selenium, a remote farm) into IPageLoader. The queue lives in one process and its state file, not in a distributed store. Proxies come from your list: it does not buy or find them for you.

Simple, one-time pricing

Free for individuals and small companies. Larger organizations pay once and use it forever.

Community

Individuals and organizations under USD 1M annual revenue and under 100 employees.

$0 free forever
  • Commercial use included
  • All features, no limits
  • All future versions
  • Community support on GitHub
Install from NuGet

No registration, no license key.

Questions

Do I need a commercial license?

Only if your organization, together with its affiliates, has USD 1,000,000 or more in annual revenue, or 100 or more employees and contractors. Everyone else uses it free, including in commercial products.

Do I need a license for z3l.FastHtmlParser too?

No. The scraper parses pages with z3l.FastHtmlParser; it is free under the thresholds, and above them the same USD 1,000 commercial license covers it, along with every other z3l library. You pay once.

Does it run JavaScript?

No. The built-in loader is HttpClient. For pages built by scripts, implement IPageLoader with Playwright, Selenium or a remote browser farm; the queue, checks and retries stay the same.

What happens if my process crashes?

With StatePath set, the run state is saved every 30 seconds and when the run stops. The next run continues: pages being loaded and items you had not taken yet are done again, nothing is lost.

Does it obey robots.txt?

When you turn on RespectRobotsTxt: robots.txt of each host is read once, disallowed pages are skipped and its Crawl-delay spaces the loads. Scraping within the law and the sites' terms is up to you.

Does the library phone home or check a license key?

No. There is no license key, no telemetry and no network access other than the pages you load.

Point it at a site

Install the package and run the quick start against books.toscrape.com, a sandbox made for scraping practice.

> dotnet add package z3l.Scraper --prerelease