← Back to blog

Crawl Budget for Research: How to Plan One That Holds

November 21, 2026
Crawl Budget for Research: How to Plan One That Holds

For a research project, a crawl budget is a set of hard limits you choose before the first request: how many pages, how deep, how often to refresh, how fast per site and how much you will spend. Search engines assign a crawl budget to your site. You assign one to the sites you study. Set the limits in the crawler itself, because a plan that lives only in a spreadsheet will not stop an unbounded crawl. Gyrence's Traverse primitive crawls outward from a starting URL, and spending caps on the account stop the bill at a number you chose.

Gyrence
Keep Research Crawls Within Budget
Gyrence pairs site traversal with account spending caps, helping you set limits before a research crawl runs.
Visit

TL;DR

  • Budget in four units: pages per domain, crawl depth, requests per second per host and dollars per run.
  • Split the budget between refreshing pages you already hold and discovering new ones. Refresh volatile sources more often than static ones.
  • Pace by the target's signals: slow down or stop on HTTP 429, repeated 403s and 5xx errors.
  • Track outcomes, not just totals. A run that returns 100% HTTP 200 can still store block pages as data.
  • Put a spending cap on the workspace before you schedule anything recurring.

Table of Contents

What "crawl budget" means outside SEO

The term comes from search. Google's documentation defines a site's crawl budget as the URLs Googlebot can and wants to crawl. It has two parts: a capacity limit, which is how much load the server can take, and demand, which depends on site size, update frequency and quality. Capacity drops when the site slows down, returns 5xx errors or sends HTTP 429 ("too many requests") responses.

That model transfers to research crawling almost unchanged, with one swap. You are the crawler, so you own both halves:

HalfIn Google's modelIn your research crawl
CapacityHow much the server toleratesHow fast each target lets you go, plus your own cap on requests and spend
DemandWhat is worth crawlingWhich pages answer your research question, and how often they change

The target's tolerance is not yours to negotiate. It shows up as latency, 429s and 403s. Your job is to read those signals and back off.

Step 1: Size the scope in pages, not domains

Start from the question, not the site. A compliance team tracking disclosures needs filing pages and nothing else. A market-intelligence team needs product, pricing and changelog pages. Write the answer as a list of URL patterns.

Then put numbers on it:

  1. Estimate page count per source. Map the domain first. Gyrence's Map primitive reads a domain's URL graph via sitemaps, so you see how large a source is before you pay to fetch it. Sitemaps have limits of their own; see what to do when sitemaps hit 50,000 URLs.
  2. Set a hard page cap per domain. Pick a cap that covers your pattern list with margin. Without one, a calendar or filter URL can generate pages forever.
  3. Set a depth limit. Most useful content sits within a few clicks of a known entry point. Our guide to crawl depth limits covers safe defaults.
  4. Exclude URL traps. Sort parameters, session IDs and calendar pages multiply URLs without adding information. URL deduplication saves bandwidth and credits in one move.

Multiply pages by the number of refreshes and you have your request volume. Everything else is a rate and a price on that number.

Step 2: Decide how often each source needs a refresh

Freshness costs requests. Research on index freshness frames it as a trade-off: each extra crawl of a page lowers the chance your copy is stale, but costs the same as any other crawl. The NISS paper on keeping a search index fresh models page changes as a Poisson process and derives crawl intervals from the relative cost of a stale page versus a crawl.

You do not need the math to use the idea. Tier your sources:

  • Fast-changing (news, pricing, status pages): refresh daily or more.
  • Slow-changing (documentation, policy pages): refresh weekly or monthly.
  • Static (archived filings, past reports): fetch once, store with a timestamp.

Crawling everything at the pace of your most volatile source is the easiest way to blow the budget. Adapt from what you observe: if a page has not changed in several checks, check it less often.

This is also where crawlers divide capacity between revisiting known pages and finding new ones. A look at refresh versus discovery in crawlers shows the same split from the site owner's side. Set it explicitly: a monitoring project leans toward refresh, a landscape study toward discovery.

For pages you only need to know changed, polling the full page is wasteful. WebDoppler monitors pages and sends webhook alerts, so you refetch when something moved instead of on a blind timer. Our priority URL scheduling post explains how crawlers decide what to fetch next.

Step 3: Set a polite rate per host

The budget has a speed limit, and it belongs to the target. Published guidance gives starting points:

Translate that into a budget line. At one request every five seconds, a host yields 17,280 requests a day. A 10,000-page cap fits inside one day of crawling. At one request every 15 seconds, the same cap takes under two days. Polite pacing rarely blocks a research project. It mostly sets the calendar.

Retries count against the budget too. Keep them inside the same rate limit as first attempts, and back off after a 429 rather than retrying immediately. Our posts on polite crawling defaults, rate limits and backoff and robots.txt compliance cover the details.

Step 4: Convert the plan into dollars

For developers burned by metered bills, this is the step that matters. Work it out as a formula:

total calls = pages per cycle × refreshes per period × sources

Then multiply by the cost per call. Gyrence bills in credits per call, with predictable per-call cost and no separate charge for AI extraction. Extraction is bundled, so a run that fetches and extracts does not add a second meter.

A worked example: 40 sources, each capped at 500 pages, refreshed weekly, is 20,000 page fetches per cycle. A monthly plan is easy to compare against that number, and a cap guarantees you cannot go past it by accident. If you would rather pay only for what you use, Pay-As-You-Go is metered by credit.

Weekly page-fetch estimate from source caps

Crawling costs also include what you store. If a fetch returns a block page and your pipeline saves it as data, you pay for the request and corrupt the dataset. Every Gyrence call returns a typed response, failure cases included, so a block shows up as a failure and not as content. Why scrapers store block pages as data walks through it.

Step 5: Monitor, then revise

A budget is a hypothesis. Check it against each run.

  • Outcomes by status, per host. Rising 429s mean your rate is too high. Rising 5xx errors can mean you are straining the server. Google's own model lowers capacity on exactly these signals.
  • New URLs per page fetched. If this ratio stays flat or rises late in a crawl, you are probably in a URL trap, not finding content.
  • Duplicate rate. High duplicates mean your normalization or exclusions need work.
  • Content checks. Assert that fetched pages contain what you expect, so a soft block does not pass as success.
  • Usage against the cap. Archive-It's guidance on monitoring a data budget suggests reviewing scheduled crawl history, and when a crawl shows a sudden jump in new data, checking the report, adjusting scope and running a test crawl before the next scheduled run.

If your work needs an audit trail, keep logs of what you fetched and when. The Alliance guidelines recommend retaining archived query logs for at least three years. Source-traceable research depends on being able to say which URL produced which record, on which date.

Revise the plan when a source changes structure, when error rates climb or when the question changes. Drift detection helps with the first of these: you hear about a changed page layout from an alert, not from a broken dataset.

Our view

We built Gyrence around a simple observation: web data is a storm, and most crawl budgets are written for calm weather. A plan that assumes every request succeeds will overspend the first time a site rate-limits you or serves a challenge page.

So we treat failure as data. Search, Traverse, Fetch, Extract and Map all return typed, discriminated-union responses, including the failure cases, which means your agent or pipeline can branch on a 429 or a blocked page instead of guessing. Spending caps sit on top, so a misconfigured crawl hits a ceiling you chose and not a surprise invoice.

Our extraction is bundled with the call, with no separate AI charge, because the budget you plan is the budget you should pay. For a research team, the point of a crawl budget is not to crawl less. It is to know, before a run starts, what it can cost and what it can return.

Glen

Run your research crawl on Gyrence

Gyrence gives you the pieces this plan needs in one API, also reachable through a hosted MCP endpoint for agent builders working in LangChain, LlamaIndex or a custom client. Map a domain, crawl it with Traverse, fetch pages as clean markdown, extract structured JSON with a prompt or schema and watch changes with WebDoppler. Plans run from a Free tier and Pay-As-You-Go to Standard at $75 per month, Growth at $299 per month and Scale at $549 per month. See the pricing page to match a plan to your page count, or look at WebDoppler if your project is about change.

Gyrence

FAQ

Is crawl budget only an SEO concept?

No. Google uses it to describe how much of your site its crawler will fetch. In research, the same idea works in reverse: it is the limit you set on how much of someone else's site you will fetch, how fast and at what cost.

How many pages should I cap per domain?

Start from the URL patterns your question needs, estimate their count from a sitemap or a Map pass, and add margin. A cap that fits your patterns and no more keeps a trap from consuming the budget. Raise it only after reviewing a completed run.

How fast should a research crawler go?

Start slow. AWS guidance suggests one request every 10 to 15 seconds for small or medium sites, and the Alliance for Responsible Data Collection suggests roughly one page every five seconds. Slow down or pause when you see 429 responses, and stop if 403s persist.

How do I stop a crawl from costing more than planned?

Set limits in three places: pages per domain, depth and a spending cap on the account. Then watch the ratio of new URLs to pages fetched, which flags a runaway crawl early.

Sources