← Back to blog

Idempotency Keys for Scraping: Retry Without Double Billing

November 14, 2026
Idempotency Keys for Scraping: Retry Without Double Billing

An idempotency key is a unique ID your client attaches to one logical scraping request, so that when a timeout forces a retry, the server can recognize the repeat and return the first outcome instead of doing the work, and charging for it, again. Generate the key once per operation, persist it, resend it unchanged on every retry, and treat any provider's replay window and credit rules as part of your cost model.

That matters most if a surprise bill has already burned you. A scraper that retries blindly after a timeout can pay twice for one page. A scraper that retries with a key, and an agent loop that knows when to stop, pays once.

Gyrence
Keep Scraping Costs Predictable
Gyrence gives web data teams spending caps and typed failure responses for scraping workflows that need clear outcomes.
Visit

TL;DR

  • One key per logical operation (one URL, one extraction job, one tool call), never per HTTP attempt.
  • On retry, resend the same key with the identical method, path, query and body. Change the request and you need a new key.
  • Keys expire. Documented windows are commonly 24 hours, and some gateways use minutes. Never rely on a key outside the window.
  • A replay is not always free: some providers still consume credits on replayed responses, others do not. Read the docs before you budget.
  • Idempotency does not replace backoff, retry limits or spending caps. It works alongside them.

Table of Contents

What an idempotency key does for a scraper

Most scraping calls look read-only, so it is tempting to assume retries are harmless. They are not, for two reasons. The call may start a job (a crawl, an extraction, a monitor) that has side effects. And the call may be metered, so each execution costs credits.

The AWS Builders' Library states that APIs with side effects are not safe to retry unless they provide idempotency, meaning the side effects happen only once no matter how many times you retry. It also notes that read-only APIs are typically idempotent while resource-creation APIs may not be.

The mechanism is simple. Per MDN's documentation of the Idempotency-Key header, the client sends a unique key with a POST or PATCH request. For a new key the server does the work and stores the key. For a repeat it skips the work and responds as though it had done it. A client that never received an acknowledgment can therefore resend safely.

Google Cloud's guide to idempotency puts the same idea in one line: repeated applications produce the same final effect with no extra side effects. That is the property you want when a network drops the response but the server already did the work.

What an idempotency key does for a scraper

Generating and reusing keys

The common failure is generating a new key on every attempt. That defeats the whole mechanism, because the server sees each retry as a new operation.

The rule, as Stitch's idempotency docs describe it: generate a new key for each logical operation, store it with the operation, and retry an ambiguous request with the same key and the exact same method, path, query parameters and body. Generate a new key only for a new logical operation. Stitch recommends UUIDs created once and persisted with the operation.

For scraping, define "logical operation" deliberately:

OperationKey scopeNew key when
Fetch one URL for a snapshotURL + snapshot run IDThe next scheduled run starts
Structured extractionURL + schema version + run IDYou change the schema or prompt
Site crawlStart URL + depth/limit settings + run IDYou change the crawl settings
Agent tool callAgent task ID + step numberThe agent deliberately re-asks

The run ID in each row is what separates "retry the same job" from "collect again tomorrow." Without it, a key tied only to the URL would replay yesterday's page when you wanted today's.

Persist the key before you send the request. If your worker crashes after sending but before saving, you lose the key and the ability to retry safely. Write the key to your job table first, then call the API.

What the server should do with duplicates

Whether you are consuming a provider's implementation or building a thin layer of your own, the same cases come up.

Same key, same request, finished. Return the stored status and body. CaptainDNS documents a replay header, X-Idempotent-Replay: true, so the client can tell a replay from a fresh result.

Same key, request still running. MDN notes a server may return 409 while the original is still processing. Your client should wait and retry with backoff, not generate a new key.

Same key, different payload. MDN describes an "idempotency fingerprint," a hash of all or part of the request stored with the key, and a server may return 422 if the fingerprint differs. Clemta's docs bind the key to method, path, body and API version and refuse a mismatch. Treat this error as a bug in your code: you reused a key for something new.

Server error with unknown outcome. Clemta treats a 5xx as not a settled outcome and releases the key so a retry runs again, while replaying both successes and rejections. The MCP Runtime gateway takes the opposite, cautious position for tool calls: per its design record, it never re-executes a tool after an ambiguous 5xx. It stores a failed-unknown marker and returns 409 idempotency_outcome_unknown on retry. Know which behavior your provider has, because it decides whether your retry might run the job twice.

Four outcomes for duplicate idempotency keys

Key lifetime and what a replay costs

A key is not forever. Stitch states keys expire 24 hours after the first request, and Clemta remembers keys for 24 hours. The MCP Runtime gateway uses a 5-minute window and scopes keys to the verified principal, session and server. Retrying after the window means the server may treat your request as new.

Then there is billing, where providers genuinely differ:

  • Talonic states a duplicate inside the window returns the cached response with no additional credit cost.
  • CaptainDNS states that replays skip processing but credits are still consumed.

So "idempotent" does not automatically mean "free to retry." Put the replay-cost rule from your provider's docs in your cost model, then add a hard ceiling on top. A key protects you from double execution. It does not cap how many times your loop calls.

What idempotency keys do not give you

Two misconceptions cause most duplicate-data bugs.

A key is not content deduplication. Flexprice's glossary points out that keys do not prevent duplicate work when keys differ, and that protection ends at the provider's retention window. If two workers generate different keys for the same URL, you get two fetches.

A key does not make your pipeline exactly-once. The server-side guarantee covers the API call. Your own database writes and downstream alerts are separate. Put a unique constraint on your observation identity (normalized URL plus run ID plus parser version, for example) and write results with an upsert, so a replayed response lands on the same row. Any webhook or queue message you emit after the write needs its own deduplication on the consumer side.

When to stop retrying

Keys make retries safe. They do not make them wise. The AWS Well-Architected guidance (REL05-BP03) recommends exponential backoff, jitter to randomize intervals, and a limit on the maximum number of retries so the backend gets time to recover.

Apply that in a scraper with a simple decision table:

  1. Timeout or connection reset: retry with the same key, backoff plus jitter.
  2. Rate limit response: wait for the signaled delay, then retry with the same key.
  3. Permanent failure (not found, blocked by policy, schema mismatch): stop. Record the failure and move on.
  4. Retry budget exhausted: park the job in a dead-letter queue for inspection.

The branching only works if the response tells you which case you are in. That is why typed failures matter: a response that says what went wrong lets code decide, where an opaque error forces a blind retry. For the mechanics, see our posts on exponential backoff for scraping and scraping retries, and this walkthrough of scraper retry logic with jitter and a global deadline. For the concept in API-design terms, read idempotent web fetch explained. If your errors are the problem, scraping API errors covers the checks.

Idempotent scraping with agents and MCP

Agents retry on their own. A model that sees a timeout will often call the same tool again, sometimes with slightly reworded arguments. That is the worst case for keys: a reworded request is a different request.

Two habits help. First, have the orchestrator, not the model, own the key: derive it from the task ID and step, and inject it into the tool call. Second, return failures to the model as data it can reason about ("outcome unknown, do not re-run") rather than a raw exception. The MCP Runtime design above shows the principle: after an ambiguous failure, the safe answer to a retry is an explicit conflict, not a second execution.

Where Gyrence fits

Gyrence's Search, Traverse, Fetch, Extract and Map calls return typed, discriminated-union responses, failure cases included, so your retry logic can branch on what actually happened. Billing is credit-based with predictable cost per call and spending caps, and there is no separate AI extraction charge, so a retry loop has a ceiling and a known unit price. The Standard plan is $75 per month, Growth is $299 per month and Scale is $549 per month; Free, Founders and Pay-As-You-Go are also listed on the pricing page.

Gyrence

If you track pages for change, WebDoppler monitors them and sends webhook alerts. Webhooks are a downstream side effect, so deduplicate them on your consumer with your own key, as described above.

Our view

Retries are where scraping bills go wrong. Nobody plans to pay twice for a page, but a timeout, a loop and an agent with no memory will do it for you.

We build for that case. Every primitive returns a typed result, failures included, because a retry decision is only as good as the signal behind it. Spending caps exist so that when your logic is wrong, your bill is still bounded. Predictable cost per call means you can compute the worst case before you ship.

Our advice: key every logical operation, persist the key first, cap retries, and read the replay-cost rule of every API you depend on. Honesty about failure modes is the point of the design.

Glen

FAQ

Should I generate a new idempotency key on every retry?

No. A new key per attempt makes each retry look like a new operation, which is exactly the duplicate you were trying to avoid. Generate the key once per logical operation, store it, and reuse it with an identical request until you get a final answer.

Do idempotency keys make retries free?

Not necessarily. One provider documents that replays within 24 hours cost no extra credits, while another documents that credits are still consumed on replay. Check your provider's docs, and use a spending cap as the backstop.

What should my key be built from?

A random UUID persisted with the job works. If you want deterministic keys, derive them from the fields that define the observation: normalized URL, run ID, and schema or parser version. Leave out the attempt number, and include the run ID so tomorrow's collection is not mistaken for today's.

What happens if I reuse a key with a different request?

A well-built server rejects it. MDN describes a 422 response when the stored fingerprint differs, and some APIs return a named mismatch error. Fix it by issuing a new key for the changed request.

Sources