← Back to blog

Web Scraping Failure Modes: A Field Guide for Developers

November 17, 2026
Web Scraping Failure Modes: A Field Guide for Developers

Scrapers rarely fail loudly. The worst failures return HTTP 200 with a wrong, empty or stale payload, and your pipeline treats it as truth. The fix is to classify every outcome into a typed result, retry only what is retryable, cap what a retry loop can spend, and validate what comes back. Gyrence returns typed, discriminated-union responses for every call, failure cases included, so your code branches on the outcome instead of guessing.

Gyrence
gyrence.com
Make Scraping Failures Explicit
Gyrence returns typed outcomes, including failure cases, so your code can handle scraping errors without guessing.
Visit

TL;DR

  • Most scraping failures fall into five buckets: blocked, transient, missing, changed and silently wrong.
  • Silent failures (empty fields, shifted selectors, stale pages) cost more than loud ones, because nothing alerts you.
  • Retry only retryable errors, with bounded backoff, and honor Retry-After when a server sends it.
  • Put a hard spending cap around every retry loop, or one bad target becomes a bad invoice.
  • Treat "failed" as a first-class result your agent can reason about, not as an exception to swallow.

Table of Contents

The five failure modes that matter

A useful taxonomy is one your code can branch on. These five cover nearly everything a scraper or agent meets.

ModeWhat you seeRetry?What to record
Blocked403, 429, challenge page, CAPTCHASometimes, after a delayStatus, delay hint, target
TransientTimeout, 502, 503, connection resetYes, boundedAttempt count, elapsed time
Missing404, 410, redirect to home pageNoTerminal state, URL
Changed200, but layout or schema movedNo, fix extractionField-level diff
Silently wrong200, plausible but empty or stale dataNo, flag for reviewValidation failure reason

The first three announce themselves. The last two are the ones that corrupt a dataset.

Blocked requests

A block is a site telling you to slow down or go away. The standard signal for rate limiting is HTTP 429, defined in RFC 6585. Servers can attach a Retry-After header, which RFC 9110 defines as the minimum wait before a follow-up request. If it is present, honor it. Hammering a 429 only lengthens the block.

Not every block is a status code. Many sites return a 200 with a challenge page or a login wall. If your parser sees a page with no product and no error, that is a block your status-code check missed. Check content, not only status.

Before you debug a "block", confirm the path is allowed at all. A robots.txt checker tells you whether a crawler is permitted on a URL, which separates a policy refusal from a technical failure. For rate-limit handling in detail, see our write-up on scraping rate limits.

Blocked requests

Transient failures: timeouts and 5xx

Timeouts, 502s and 503s usually clear on their own. These are the only failures where a retry is the right default. The rules:

  1. Cap attempts. Three to five is typical. Unbounded loops are how a single dead host eats a budget.
  2. Back off exponentially and add jitter, so parallel workers do not retry in lockstep.
  3. Prefer the server's Retry-After over your own schedule when it sends one.
  4. After the last attempt, record a terminal failure. Do not loop, and do not return an empty result that looks like success.

The mechanics, including three jitter modes, are in our guide to exponential backoff for scraping, and the broader pattern is in scraping retries explained.

Missing pages

A 404 or 410 is an answer, not an error to retry. The trap is soft 404s: the server redirects a dead product URL to the category page or home page and returns 200. Compare the final URL to the requested one and treat a surprise redirect as "missing". Record it as terminal so your pipeline does not re-request it every night.

Changed pages

Sites change layouts, rename classes, move content behind JavaScript, and swap A/B variants. A selector-based scraper breaks the moment the markup moves, and it often breaks quietly: the selector matches nothing, the field comes back null, and the row still gets written.

Two defenses help:

  • Schema-guided extraction. Define the fields you expect and their types. A prompt or schema describes meaning ("the price") rather than a DOM path, so cosmetic changes break less often. Our post on structured data extraction with schema validation covers the pattern.
  • Drift monitoring. Watch the pages you depend on and alert when they change, rather than discovering it in next month's report. WebDoppler does this with webhook alerts.

Silently wrong data

This is the mode that hurts compliance and financial-research workflows most, because a wrong number with no source trail is worse than no number. Run two validation layers on every record:

  • Structural. Are required fields present, and are they the right type? A price encoded as an object or an empty string fails here.
  • Semantic. Is the value plausible? A price that falls 90% overnight, a date in the future, or a currency you have never seen should go to review rather than into the database.

Give every rejection a stable reason code (for example, a missing price or an unknown currency) so you can count failures by cause over time. A rising count of one code is a trend you can act on. A pile of generic "errors" is not.

Keep the source URL and fetch time on every record. When someone asks where a number came from, you want one lookup, not an investigation.

Why agents make failure modes worse

A human sees a CAPTCHA and stops. An LLM agent often sees an empty string and continues, sometimes inventing the missing content. Three habits prevent this:

  • Return failures to the agent as structured data it can read (what failed, whether it is retryable), not as a thrown exception that the framework summarizes into nothing.
  • Give jobs stable IDs and terminal states (completed or failed), so an agent never polls forever or runs duplicate work.
  • Put a deadline and a spend ceiling on each task.

We list more of these traps in common AI agent scraping mistakes and the agent-safe scraping guide. If your agent connects over the Model Context Protocol, see MCP web scraping.

Failure modes in your bill

Retries are failures with a price tag. A site that blocks you at 3 a.m. can trigger thousands of retried calls, and on usage-priced platforms every one is billed. Surprise invoices come from three places: unbounded retry loops, separately metered extraction add-ons, and crawls that follow links with no ceiling.

Three causes of surprise scraping costs

The controls are boring and effective: a per-workspace spending cap, a maximum attempt count, a crawl depth and page limit, and a single predictable cost per call. Decide these before the first production run, not after the first bad month.

A short checklist

  1. Does every call return success or a typed failure, never a bare null?
  2. Do you separate retryable from terminal failures?
  3. Is every retry loop bounded in attempts, time and spend?
  4. Do you validate structure and plausibility before storing?
  5. Does each record carry its source URL and fetch time?
  6. Is something watching for page drift?

How we think about failure

Most scraping vendors show you the happy path. The 200 with a clean body is easy to sell. The hard part is everything else, so we built Gyrence around it. Every one of the five primitives (Search, Traverse, Fetch, Extract, Map) returns a typed response where the failure cases are first-class members of the union.

We think that is the only honest design for agents. An agent can reason about a failure it can see. It cannot reason about a silence. Spending caps and predictable per-call credits follow from the same belief: a failure should cost you a known amount, not an open-ended one. Extraction is bundled, with no separate AI extraction charge, so your bill does not have a second meter hiding in it.

Glen

Run it with Gyrence

Gyrence is a web data API with five composable primitives plus a hosted MCP endpoint. It returns typed responses including failures, and billing runs on credits with spending caps. Plans on the pricing page include Free, Pay-As-You-Go, Standard at $75 per month, Growth at $299 per month and Scale at $549 per month.

Gyrence

If drift is your main worry, start with WebDoppler. To try the API itself, begin at gyrence.com.

FAQ

What is the most common failure mode in web scraping?

Silent failure is the most common and the most expensive: the request succeeds, but the data is empty, stale or mis-parsed. Loud failures such as timeouts and 429s are easier to catch because they produce a status code. Validate structure and plausibility on every record to catch the quiet ones.

Should I retry every failed scrape?

No. Retry transient failures such as timeouts and 5xx responses, with a bounded number of attempts and exponential backoff. Do not retry 404s or extraction failures caused by a changed layout. For a 429, wait at least as long as the Retry-After value if one is sent.

How do I stop retries from inflating my bill?

Cap attempts per URL, cap total spend per workspace, and limit crawl depth and page count. Choose a provider whose cost per call is fixed and whose extraction is not metered separately, so a failed run has a known maximum cost.

How should an AI agent handle a failed fetch?

It should receive the failure as structured data stating what failed and whether it is retryable. It can then retry, skip or escalate instead of guessing. Give each task a deadline and a terminal state so it never polls indefinitely.

Sources