Scraping Retries Explained: A Developer's Playbook

Retry only the failures that are actually transient, back off exponentially with jitter, and cap your attempts. That's the entire policy in one sentence, and most scraping outages trace back to violating one clause of it.

The reason this matters: a scraper that retries everything blindly turns a five-second network hiccup into a five-minute hang, and a scraper that never retries anything turns normal internet noise into missing data. Neither failure mode is acceptable in production. The trade-off is resilience against upstream flakiness versus politeness toward the target server, and a correct retry policy sits precisely at the point where you recover from real transient errors without hammering a site that's already telling you to slow down.

Here's the minimum viable checklist before you write another line of scraper code:

• Retry: connection timeouts, DNS failures, 502/503/504, 408, and 429 (with special handling).
• Don't retry: 400, 401, 403, 404, 422, and anything else that signals the request itself is wrong.
• Back off: exponential growth with randomized jitter, never a fixed delay.
• Cap it: a hard maximum on attempts (3 to 5 is standard) and a total wall-clock timeout for the whole operation.
• Watch it: log every attempt, the delay before it, the response code, and whether a Retry-After header was present.

Key Takeaways

A correct retry policy retries only transient failures, backs off exponentially with jitter, honors server-issued Retry-After headers exactly, and caps both attempts and total wait time.

Table of Contents

• What Are Request Retries and Why Do They Matter for Scrapers?
• Which Failures Are Worth Retrying?
• Backoff Strategies and Jitter: How Long Should You Wait?
• Idempotency: When Is It Actually Safe to Retry?
• Respecting Rate Limits: How to Handle Retry-After, 429, and 503
• Retry Limits, Caps, and Circuit Breakers
• How to Implement Retries in Python, Scrapy, and JavaScript
• Observability, Testing, and Common Retry Mistakes
• Designing Failure-Aware APIs: Lessons for Building Reliable Systems
• Quick Checklist: Implement Retries Safely
• What Building Resilient Scraping Systems Actually Teaches You
• An Alternative: Skip the Retry Logic Entirely
• Sources
• FAQ

What Are Request Retries and Why Do They Matter for Scrapers?

A request retry is an automated re-attempt of a failed HTTP call, governed by a retry policy: rules that define which failures qualify, how long to wait between attempts (the backoff), how much randomness to add to that wait (jitter), whether the operation is safe to repeat (idempotency), and how many total attempts are allowed (the retry budget).

Retries exist because the open web is inherently unreliable. DNS resolution occasionally fails. TCP connections drop mid-handshake. Origin servers return 503 during deploys or traffic spikes. Rate limit windows reset on a rolling basis, so a request that fails at second 59 often succeeds at second 61. Without retries, every one of these transient blips becomes a permanent data gap in your scraping pipeline, and at scale, transient blips are not rare events. A scraper pulling from thousands of domains will see network flakiness constantly, just distributed unevenly across targets.

Retries are a reliability primitive, the same category of tool as a timeout or a circuit breaker, not a workaround for bad scraper design. That said, they come with real costs:

• Pros: higher successful-fetch rate, resilience to normal internet jitter, fewer false negatives in your dataset, graceful handling of rate limit windows.
• Cons: added latency per failed request, risk of amplifying load on an already-struggling server, and if misconfigured, the potential to look like an attack (which gets you IP-banned faster than a single failed request ever would).

Get the balance wrong in either direction and you either lose data or lose access. The rest of this piece is about landing in between.

Which Failures Are Worth Retrying?

Not every failure deserves a second attempt. The general principle from ProxiesAPI's retry guidance is to classify failures into buckets, transient network issues, temporary upstream errors, permanent client errors, and soft blocks, then apply a different action to each bucket rather than a single blanket rule.

Server errors in the 5xx range generally mean the problem is on the target's end and might resolve itself. Client errors in the 4xx range usually mean your request is malformed, unauthorized, or pointed at something that doesn't exist, and retrying won't fix any of that. Azion's breakdown of HTTP status codes confirms this pattern: 408, 429, 502, 503, and 504 are typically treated as retryable, while most 4xx codes are not.

A couple of codes need nuance. 502 and 504 are often safe to retry only on idempotent methods (GET, HEAD), since you can't always be sure the origin server didn't partially process a POST before the gateway timed out. And a 403 is usually permanent, but if you're seeing it intermittently on the same endpoint that returned 200 a minute ago, that's frequently a soft block (a bot-detection challenge disguised as a normal response) rather than a true authorization failure. Soft blocks deserve their own detection logic, since a 200 status with a CAPTCHA page in the body is a silent failure that a naive retry policy will miss entirely, as covered in more depth in this look at why HTML parsing fails at scale.

For network-level exceptions rather than HTTP status codes: a ConnectionError or read timeout is almost always worth one or two retries, since it usually means a packet got dropped somewhere. A DNSLookupError that repeats after two attempts, though, likely means the domain is genuinely unreachable, and further retries just burn time.

Backoff Strategies and Jitter: How Long Should You Wait?

Use exponential backoff with randomized jitter, and cap the maximum delay. A typical series looks like 1s, 2s, 4s, 8s, 16s for the base exponential curve, with jitter randomizing each of those values by up to 50% so retries from different requests don't all land on the same second.

Four approaches show up in practice, and only one of them is actually recommended for production scraping:

• Constant delay: wait the same fixed interval every time. Simple, but wasteful on quick-resolving errors and useless against sustained outages.
• Linear backoff: wait grows by a fixed increment each attempt (1s, 2s, 3s). Better than constant, but still too slow to back off from aggressive rate limiting.
• Exponential backoff: delay doubles (or multiplies by some factor) each attempt. Handles both quick blips and longer outages reasonably well.
• Exponential backoff with jitter (recommended): same exponential curve, but each delay gets randomized within a range. This is the approach Scraping Central's guide to timeouts and retries recommends for production scrapers, and it's the industry default for good reason.

The jitter matters more than people expect. Without it, if you have a thousand workers that all failed on the same request at the same moment (a common scenario when a target site goes down briefly), they'll all retry at exactly 1s, then all retry again at exactly 2s, and so on. That synchronized retry pattern, sometimes called a thundering herd, can hit a recovering server with a bigger spike than the original traffic that caused the outage.

Reasonable defaults: a base interval of 1 second, a multiplier of 2, a cap around 30 to 60 seconds per individual wait, and full jitter (randomizing the entire delay range rather than just adding a small offset to a fixed value).

Here's the shape of the logic in pseudocode:

function get_backoff_delay(attempt, base=1, cap=60):
    exponential = min(cap, base * (2 ** attempt))
    jittered = random_uniform(0, exponential)
    return jittered

function retry_request(request, max_attempts=4):
    for attempt in range(max_attempts):
        response = send(request)
        if response.status == 429 or response.status == 503:
            if response.has_header("Retry-After"):
                wait(parse_retry_after(response.header))
                continue
        if is_retryable(response):
            wait(get_backoff_delay(attempt))
            continue
        return response
    raise MaxRetriesExceeded()

Pro Tip: Pick a jitter range wide enough that two workers retrying the same failure land at least a second apart on average, but not so wide that a request which should recover in 4 seconds might wait 40. Full jitter (0 to the exponential ceiling) works for most scraping workloads; decorrelated jitter, where each delay is randomized based on the previous one, is worth the extra complexity only if you're running thousands of concurrent workers against the same host.

Idempotency: When Is It Actually Safe to Retry?

GET and HEAD requests are idempotent by definition; calling them multiple times produces the same result as calling them once, so retrying them freely is safe. POST, and sometimes PATCH, are not automatically idempotent; retrying a POST that already succeeded, but timed out before the response arrived, can create a duplicate record, double-charge a payment, or trigger a webhook twice.

Most web scraping is read-heavy, so this concern applies less to the fetch itself and more to anything your pipeline does with the extracted data: writing to a database, calling a downstream API, or triggering a notification. If your scraper's retry logic wraps the entire pipeline rather than just the HTTP fetch, you need to think carefully about what happens on a duplicate run.

A few patterns make non-idempotent operations safe to retry:

• Idempotency-Key header: attach a unique key to each logical operation; the server deduplicates any request sharing that key, even if it arrives twice.
• Server-side dedupe: the receiving system checks for an existing record with the same identifier before processing, rejecting duplicates silently.
• Confirm-then-retry: for anything with irreversible side effects (sending an email, submitting a form), poll for a status result first; only retry if you can confirm the original attempt never completed.
• Background job pattern: submit the write as an async job with a return token, then poll the token's status rather than retrying the submission itself.

The API Design Guide on request retries frames this as a contract: servers that support idempotency keys make it dramatically easier for clients to retry safely, because the burden of deduplication shifts from "hope the client doesn't double-fire" to "the server guarantees it won't double-process." If you're designing a scraping pipeline that writes extracted data downstream, adding an Idempotency-Key header (often a hash of the source URL plus a timestamp bucket) to your write calls closes off an entire category of duplicate-data bugs.

Respecting Rate Limits: How to Handle Retry-After, 429, and 503

If a response includes a Retry-After header, honor it exactly rather than falling back to your own backoff math. RFC 2616 specifies that servers may include this header on temporary failures like 503 to tell clients precisely when it's safe to try again, and ignoring that explicit signal in favor of your own guess is both impolite and slower for you, since the server knows its own recovery timeline better than your exponential curve does.

The header comes in two valid forms, and your parser needs to handle both:

Numeric seconds is the safer format to expect and produce, since RFC 2616's Retry-After guidance notes that HTTP-date forms introduce clock-skew risk between client and server. If you're the one designing an API (more on that below), always prefer emitting delta-seconds.

429 and 503 deserve special treatment beyond the standard retry logic:

• On 429: treat it as a signal to slow down globally, not just retry the one request. Reduce concurrency against that host, extend your baseline backoff interval, and log the event distinctly from other retryable errors, since a pattern of 429s tells you something a single retryable failure doesn't.
• On 503: treat it as transient by default, especially if a Retry-After header is present, per the guidance on handling rate limits and retries. A 503 without a Retry-After header still warrants a retry, just with your standard exponential backoff instead of a server-dictated wait.
• Avoiding retry storms: if you're running concurrent workers against the same domain, coordinate them. A shared rate limiter or a per-host semaphore prevents twenty workers from all hitting a 429-throwing endpoint simultaneously, each independently deciding to retry.

Retry Limits, Caps, and Circuit Breakers

Cap every request at a fixed number of attempts, and cap the total wall-clock time the operation is allowed to take. These two limits interact: a max-attempts cap of 5 with an uncapped exponential backoff could still let a single request eat minutes of wall-clock time if the delays aren't also bounded. Scraping Central's recommended defaults land around 3 to 5 max attempts with connect and read timeouts in the range of 5 and 30 seconds respectively, a range most production scrapers converge on independently.

Beyond per-request limits, a circuit breaker protects both your scraper and the target host when failures aren't isolated incidents but a sustained pattern. The pattern has three states:

• Closed: normal operation, requests flow through and get retried per the standard policy.
• Open: after a threshold of failures against a given host (say, 10 consecutive 503s), the breaker trips. Further requests to that host fail immediately without even attempting a connection, for a cooldown period.
• Half-open: after the cooldown, a single test request goes through. If it succeeds, the breaker closes and normal traffic resumes; if it fails, the breaker reopens and the cooldown restarts.

Circuit breakers matter most in concurrent scraping setups, where without one, twenty workers might each independently retry against a dead host five times, burning 100 wasted requests and their associated backoff delays before anyone notices the host is down. A shared breaker state across your worker pool stops that waste after the first few failures instead.

Coordinate throttling at two levels: per-request (the retry policy covered above) and per-host (a shared concurrency pool or semaphore that limits how many simultaneous requests any one domain receives, independent of retry state). This becomes especially important during large-scale crawl jobs, where dozens of workers might be pulling from overlapping domains; the concurrency and coordination concerns are worth reading in more depth if you're running bulk domain crawling at scale.

How to Implement Retries in Python, Scrapy, and JavaScript

Python with requests and urllib3

The urllib3.Retry adapter, wired into requests, handles most of this out of the box:

from requests import Session
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry_strategy = Retry(
    total=4,
    backoff_factor=1,
    status_forcelist=[408, 429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD"],
)
adapter = HTTPAdapter(max_retries=retry_strategy)

session = Session()
session.mount("https://", adapter)
session.mount("http://", adapter)

response = session.get(url, timeout=(5, 30))

The allowed_methods parameter is what keeps this safe: restricting automatic retries to GET and HEAD avoids the idempotency problem entirely for the built-in adapter. If you need finer control, like custom logic per status code or integration with logging, a library like tenacity gives you decorator-based retry logic with configurable wait strategies and stop conditions.

Scrapy

Scrapy ships with retry middleware enabled by default, but the defaults are conservative and worth tuning:

# settings.py
RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 408, 429]
RETRY_PRIORITY_ADJUST = -1

The catch, as noted in practitioner writeups on Scrapy's error handling and retry logic, is that Scrapy's default middleware retries based purely on status code and doesn't inherently detect soft blocks, a 200 response containing a CAPTCHA page sails right through as a "success." You'll need a custom downloader middleware or a process_response hook that inspects the response body for challenge indicators and raises a retryable exception when it finds one.

JavaScript with axios

There's no built-in retry adapter in axios, so you wrap it:

async function fetchWithRetry(url, maxAttempts = 4) {
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      const response = await axios.get(url, { timeout: 10000 });
      return response;
    } catch (error) {
      const status = error.response?.status;
      const retryAfter = error.response?.headers["retry-after"];
      if (![408, 429, 500, 502, 503, 504].includes(status)) throw error;

      const delay = retryAfter
        ? parseInt(retryAfter) * 1000
        : Math.min(60000, 1000 * 2 ** attempt) * Math.random();

      console.log(`Attempt ${attempt + 1} failed (${status}), waiting ${delay}ms`);
      await new Promise((resolve) => setTimeout(resolve, delay));
    }
  }
  throw new Error("Max retries exceeded");
}

This same shape works whether you're calling a scraping target directly or wrapping calls into a managed API, and the pattern is worth understanding before you connect a scraping API to an AI agent, since agent workflows tend to compound retry logic across multiple chained calls.

Pro Tip: Test retry behavior against a deliberately flaky endpoint, one you control, that returns 503 for the first two calls and 200 on the third. Assert on attempt count, the actual elapsed time between attempts (to confirm backoff is really happening and not just logged), and that your logs captured every attempt distinctly. A retry policy that passes code review but was never tested against real flakiness is a policy you haven't actually verified.

Observability, Testing, and Common Retry Mistakes

Log more than you think you need to. At minimum, capture the attempt number, the final outcome, the backoff delay applied before each attempt, the response status code, any Retry-After value received, a correlation ID tying attempts together, and rolling error rates per host so you can spot a target that's degrading before it fully fails.

Testing retry paths requires more than unit tests on the backoff math:

1. Fault injection: stand up a test endpoint that fails predictably (fails N times, then succeeds) and run your real retry wrapper against it.
2. Timing assertions: verify that delays between attempts fall within your expected jitter range, not just that a delay happened.
3. Canary runs: periodically run a small subset of real traffic through your retry logic in a monitored environment to catch regressions before a full deploy.
4. Log verification: confirm that every retry attempt actually produces a log line; a retry system that fails silently is worse than no retry system, because it hides data loss behind an apparent success.

Common mistakes worth checking your own scraper against:

1. Missing timeouts. A request with no timeout can hang indefinitely, and no retry logic will ever trigger because the request never technically fails. Set both connect and read timeouts explicitly.
2. Retrying non-idempotent methods by default. If your retry wrapper doesn't distinguish GET from POST, you will eventually duplicate a write.
3. Ignoring Retry-After. Falling back to your own backoff math when the server told you exactly how long to wait is both slower for you and rougher on the target.
4. Silent retries. If failed attempts aren't logged distinctly from the final result, a request that succeeded on the fourth try looks identical in your data to one that succeeded on the first, hiding a degrading target until it fails completely.
5. No jitter. Fixed exponential delays across many concurrent workers create synchronized retry storms against a recovering host.

Designing Failure-Aware APIs: Lessons for Building Reliable Systems

Good API design treats retries as a contract between server and client, not an afterthought the client has to reverse-engineer. A server that clearly signals what's retryable, and what isn't, makes every client integration simpler and every retry policy more precise.

Four principles should guide that contract:

• Explicit retry signals. Return Retry-After on every 429 and 503, in delta-seconds rather than an HTTP-date, since numeric seconds avoid clock-skew ambiguity between server and client clocks.
• Typed failure responses. Instead of a bare status code, return a structured payload the client can parse programmatically:

{
  "status": "failure",
  "retryable": true,
  "retry_after_seconds": 15,
  "request_id": "req_8f2a1c",
  "reason": "upstream_rate_limited"
}

• Idempotency support on write endpoints. Accept an Idempotency-Key header on any POST or PATCH that has side effects, and deduplicate on the server side, as the request-retries API design guide recommends.
• Failure-surfacing over silent degradation. A 200 response that secretly contains partial or invalid data is worse than an honest 503, because the client has no signal to retry on.

Pro Tip: A retryable: true/false field in your failure payload does more for downstream reliability than almost any other single design choice, since it removes the guesswork that forces every client to independently reverse-engineer your status-code semantics. Clients that can trust a machine-readable retryability flag write dramatically simpler, and more correct, retry loops. This kind of typed, discriminated response is exactly what separates a scraping API a team can build reliable automation on from one that requires defensive parsing at every call site.

Quick Checklist: Implement Retries Safely

Paste this into a runbook or pull request template:

• Set explicit timeouts on connect and read, separately.
• Classify every error into retryable or non-retryable before writing retry logic.
• Honor Retry-After exactly when present; fall back to backoff only when it's absent.
• Apply exponential backoff with jitter, never a fixed or purely linear delay.
• Cap total attempts (3 to 5) and total wall-clock time per operation.
• Log every attempt, not just the final outcome.
• Test against a real flaky endpoint, not just unit-tested backoff math.
• Use idempotency keys on any retried write operation.

In PR review, treat any new HTTP call without a retry classification as a blocker, the same way you'd treat a missing test. In runbooks, this checklist doubles as the first diagnostic pass when a scraper starts silently dropping data: check timeouts first, then retry classification, then whether Retry-After is actually being parsed.

What Building Resilient Scraping Systems Actually Teaches You

Prevention beats recovery, almost every time. The best retry policy in the world doesn't fix a scraper that never set a connection timeout in the first place, and most "retry bugs" I've seen traced back are actually timeout bugs wearing a retry costume. Get the fundamentals right first: timeouts, error classification, and idempotency boundaries. Retries are the layer on top, not the fix underneath.

Real-world failure modes are messier than any status-code table admits. A 200 with a CAPTCHA, a 403 that's actually a rate limit in disguise, a 503 that means "we're deploying" versus one that means "we're down for six hours." No fixed table captures all of that nuance, which is why the soft-block detection and response-body inspection matter as much as the HTTP-level logic. Test against real targets, not just the happy path a status-code chart implies.

Transparency to the consumer of your data matters more than most teams admit. A scraping pipeline that silently drops failed pages produces a dataset that looks complete and isn't, and that's a worse outcome than a pipeline that surfaces gaps explicitly. Whether you're building the scraper or the API it calls, structured, typed failure information beats a clean-looking but silently incomplete result every time.

An Alternative: Skip the Retry Logic Entirely

Gyrence is the failure-aware alternative to hand-rolling retry logic across every scraper you maintain: instead of wiring urllib3.Retry, custom Scrapy middleware, and an axios wrapper into three different codebases, you call one API that already handles the backoff, the Retry-After parsing, and the soft-block detection underneath.

Every Gyrence response comes back as a typed, discriminated union, so a transient failure looks structurally different from a permanent one before your code even has to branch on a status code. That maps directly onto the practices covered above:

• Retry-After support baked in. Rate-limit backoff happens server-side, not in your client code.
• Idempotency-aware primitives. Extract and Fetch calls are designed around safe repetition, so you're not reverse-engineering deduplication.
• Structured failure payloads. Every failure mode is typed and surfaced, not hidden behind a 200 and a CAPTCHA page.
• Spending caps. No surprise bill from a retry storm running away from you at 2 a.m.

If you're tired of maintaining retry middleware across five different scraping projects, check the Gyrence docs and see what a failure-aware API primitive looks like when the retry logic is somebody else's job.

Sources

• Hypertext Transfer Protocol -- HTTP/1.1 (RFC 2616) - Status Codes (section 10)
• Retry Policies for Web Scrapers: What to Retry vs Fail Fast | ProxiesAPI Guides
• Web-automations
• Timeouts, Retries, Exponential Backoff, Static Scraping | Scraping Central

FAQ

Is web scraping legal?

Scraping publicly accessible data is broadly legal in most jurisdictions, but it depends heavily on what you scrape, how you access it, and the target site's terms of service. Retry behavior itself is a technical concern, not a legal one, but aggressive retrying that ignores rate limits can shift a borderline case toward unauthorized-access territory, so respecting Retry-After and backing off is both good engineering and good practice.

What are the best practices for API retry policies?

Classify failures before retrying, use exponential backoff with jitter, honor any Retry-After header exactly, cap total attempts at 3 to 5, and use idempotency keys on any retried write operation. A failure-aware API design, one that returns typed, machine-readable failure signals, makes all of this easier to implement correctly on the client side.

Why do people say data scraping is bad?

Scraping earns criticism when it ignores rate limits, retries too aggressively, or extracts data in ways that violate a site's terms of service, since those behaviors degrade the target server and can cross legal lines. Scraping done with proper timeouts, retry classification, and respect for Retry-After headers is a normal, widely used data-collection technique, not inherently harmful.

Do hackers use web scraping techniques?

Some malicious actors do use scraping-adjacent techniques for credential stuffing or content theft, which is part of why sites deploy rate limits and soft-block challenges in the first place. That overlap is exactly why a legitimate scraper's retry policy needs to look distinct from an attack pattern: predictable backoff, honored rate limits, and capped attempts, rather than rapid-fire retries that mimic abuse.

Does Gyrence handle retries automatically?

Yes. Gyrence's API applies backoff and Retry-After handling server-side and returns typed, discriminated failure responses, so client code doesn't need to implement its own retry middleware for common transient failures.

Recommended

• Top 10 Common AI Agent Scraping Mistakes to Avoid
• Data Collection Repeatability: A Developer's Guide
• Why HTML Parsing Fails at Scale: A Developer's Guide
• Best Article Scrapers for Developers and Data Teams