Visit Website →
← Back to blog

Adaptive Throttling for Web Scraping: How to Do It Right

November 3, 2026
Adaptive Throttling for Web Scraping: How to Do It Right

Adaptive throttling means your scraper changes its own request delay or concurrency based on what the target server does: slower when latency climbs or errors appear, faster when responses are healthy. A fixed rate ignores all of that. Done well, adaptive throttling gets you more pages per hour with fewer 429s, fewer bans and a bill you can predict.

This guide covers the algorithms, the signals to watch, the limits you should never cross, and how to cap cost when retries go wrong.

Gyrence
Keep Scraping Costs Predictable
Gyrence provides web data APIs with spending caps and structured failure responses, so your agents can handle scraping errors without guessing.
Explore Gyrence

TL;DR

  • Adaptive throttling adjusts delay or concurrency from observed latency and response codes. Fixed rates do not react to server capacity.
  • Treat 429 and 503 as hard signals: honor Retry-After, back off exponentially, and add jitter.
  • Adapt within bounds. Set a minimum delay, a maximum delay, and a ceiling taken from robots.txt and the site's terms.
  • Retries cost money on metered APIs. Put a spending cap in front of the loop, not behind it.
  • Raise concurrency gradually and watch status counts, retry counts and latency.

Table of Contents

What adaptive throttling actually does

A fixed rate says "one request every two seconds" no matter what. If the server slows down under load, you keep pushing. If it is idle, you leave throughput on the table.

Adaptive throttling closes the loop. Scrapy's AutoThrottle extension computes each download slot's delay from observed response latency, aiming for a target average number of concurrent requests per remote site, and constrains the result with bounds. A fixed delay stays constant and can produce bursts.

The Spatie crawler documentation describes the same idea with an exponential moving average: delay rises when the server responds slowly, falls when responses speed up, and never leaves the configured minimum and maximum.

The common ingredients:

  1. A signal: latency (smoothed with an EWMA), status codes, or both.
  2. A controller: a rule that turns the signal into a new delay or concurrency value.
  3. Bounds: a floor and a ceiling so the controller cannot run away in either direction.

Signals worth reading

Latency is the earliest warning. Errors are the loudest. Watch both, per domain, not globally.

Scrapy's optimization guidance recommends monitoring counts of 429 and 503 responses, ban-page responses, retry counts and download latency while you raise concurrency gradually. It also separates global concurrency, per-domain concurrency and download delay. Per-domain is the one that matters for politeness: exceeding the rate a target tolerates can cause throttling, errors or bans, and can make the crawl slower overall.

Two signals deserve special handling:

  • HTTP 429. Per MDN, it means the client sent too many requests in a given period, and the response may include Retry-After.
  • Retry-After. MDN documents it as either a number of seconds or an HTTP date, and as a minimum wait before the next request, notably on 429 and 503. Your controller should never schedule a retry earlier than this value.

A false "success" is also a signal. A 200 that returns a CAPTCHA or block page is a ban in disguise. Classify it as a failure so the controller backs off. This is why we return typed failure cases instead of a bare body: an agent or pipeline can branch on blocked versus timeout versus ok without parsing HTML.

Controllers that work

Latency-target control

The AutoThrottle approach: pick a target concurrency, measure latency, set delay to roughly latency divided by target concurrency, clamp to bounds. It is simple, stable and good for steady crawls.

AIMD

Additive increase, multiplicative decrease, the same family used in TCP congestion control. The aioscraper rate limiter implements it per group: failures multiply the interval, and after successive successes the interval shrinks by a fixed step. It can also honor Retry-After. AIMD reacts fast to trouble and recovers cautiously, which is the right asymmetry when the other side can ban you.

Backoff with jitter

When a request is throttled, retry later, and later again if it keeps failing. AWS retry guidance recommends exponential backoff with full jitter, with longer base delays for throttling errors than for transient ones. The formula is a uniformly random value multiplied by a capped exponential delay.

Jitter matters when you run many workers. AWS explains that full jitter spreads clients across the backoff window so they do not retry at the same instant and recreate the overload. The same page distinguishes adaptive retry mode, which lowers the rate of initial requests after throttling, from standard mode, which mainly delays retries. For scraping you want the adaptive behavior: backoff on one request is not enough if the next fifty go out at full speed.

It also says non-retryable errors (authorization, validation, not found) should be returned immediately. Do not spend a retry budget on a 404.

For jitter modes and cost caps in detail, see our exponential backoff guide for scraping and the retries playbook.

Backoff with jitter

A worked example

Suppose you crawl one domain with a floor of 1 second and a ceiling of 30 seconds, starting at 2 seconds.

EventController actionNew delay
5 successes, latency steadySubtract fixed step1.5 s
Latency doublesRaise delay toward latency / target concurrency3 s
429 with Retry-After: 20Pause domain 20 s, then multiply delay6 s
Block page served as 200Count as failure, multiply delay12 s
10 clean responsesStep down gradually11 s, then lower

The exact numbers are yours to tune. The shape is what counts: quick to slow down, slow to speed up, always clamped.

Staying inside the site's limits

Adaptive does not mean "as fast as the server survives". Scrapy's maintainers stress that the target site's tolerated rate, not your hardware, is the governing constraint. Practical defaults:

  • Read robots.txt and translate Crawl-delay and Request-rate into your floor delay and concurrency ceiling. Our notes on robots.txt compliance and robots meta tags cover the edge cases.
  • Use a documented API or bulk export when one exists.
  • Never rotate identities to dodge a 429. Slowing down is the fix; evasion is a different problem with legal weight. The FTC recognizes legitimate bot uses such as search crawling, monitoring and market analysis, while judging deceptive or unfair conduct case by case under the FTC Act. If your project touches personal data or contractual terms, get advice from counsel.

If you are building the server-side counterpart, this walkthrough of Redis and Lua rate limiting shows what limits look like from the other end, which helps you predict how targets will respond. For crawler politeness defaults, see polite crawling and ethical scraping with rate limits.

Throttling and your bill

On a metered scraping API, every retry can be a charge. A runaway loop, a bad deploy or a stuck agent turns adaptive logic into a spending problem.

The research here is consistent. Zuplo's API cost protection guide argues for hard caps by default on free and self-service plans, with explicit opt-in before overages on paid tiers, and says absolute quotas and spend caps protect against runaway retries and accidental loops more reliably than monitoring alone.

Practical rules:

  1. Cap before you retry. Give each job a maximum retry count and a maximum spend.
  2. Separate rate from quota. Rate limits protect capacity per second; quotas protect revenue per month. Our piece on rate limits vs quotas explains why you need both.
  3. Do not add parallelism to beat a quota. Extra workers help only when you are latency-bound, not quota-bound.
  4. Cap the backoff. A 20-second ceiling with spending caps is a sane starting point; see scraping rate limits.

Perspective

We built Gyrence around one belief: a scraper should tell you what happened, not guess. Web data is a storm, and the worst outcome is a pipeline that keeps hammering through it and bills you for the privilege.

That is why every call returns a typed, discriminated-union response, including failures. When a fetch is blocked, rate-limited or empty, your agent sees that as data it can reason about, then slow down, retry within a budget, or stop. Adaptive throttling is only as good as the signals feeding it, and hidden failure modes starve it.

Spending caps and predictable cost per call finish the job. Backoff logic will have bugs. The cap means a bug costs you a bounded amount, not a surprise invoice.

Glen

Putting it into a pipeline

A throttled web-data workflow usually runs in this order: Search to find candidate URLs, Map to list a domain's URL graph, Fetch to clean pages to markdown for RAG ingestion, and Extract to pull schema-guided JSON. Throttle at the domain level across all of them, since Map and Fetch hit the same server. If sitemaps are huge, this sitemap guide covers what to do past 50,000 URLs.

When evaluating any scraping API, including alternatives to the one you use today, ask:

  • Does it expose rate-limit and failure states as typed responses, or hide them behind generic errors?
  • Does it work with MCP clients and agent frameworks?
  • Is extraction bundled, or billed separately?
  • Can you set a hard spending cap?

Try it with a cap on

Gyrence offers a Free plan, Pay-As-You-Go metering, and paid tiers: Standard at $75/month, Growth at $299/month and Scale at $549/month. Extraction is bundled with no separate AI charge, and spending caps are part of the billing controls. Compare plans on the Gyrence pricing page. If you need to watch pages for changes and get alerts, look at WebDoppler.

Gyrence

FAQ

What is the difference between rate limiting and throttling?

Rate limiting is the cap a server enforces on you, often returning 429. Throttling is what your client does to stay under it. Adaptive throttling tunes that behavior from live feedback instead of a hard-coded number.

Should I retry every failed request?

No. Retry transient failures and throttling responses, with exponential backoff and jitter. Return authorization, validation and not-found errors immediately, since retrying them wastes time and credits.

Does more concurrency make scraping faster?

Only when the work is latency-bound and the target tolerates it. If the limit is a quota or the site's tolerated rate, extra workers produce more 429 responses and a slower crawl. Raise concurrency gradually and watch error counts.

How do I honor Retry-After in an adaptive controller?

Parse it as seconds or an HTTP date, pause that domain until the time passes, then resume at a higher delay than before. Treat it as a minimum wait, never a suggestion.

Sources