An idempotency key is a unique ID your client attaches to one logical scraping request, so that when a timeout forces a retry, the server can recognize the repeat and return the first outcome instead of doing the work, and charging for it, again. Generate the key once per operation, persist it, resend it unchanged on every retry, and treat any provider's replay window and credit rules as part of your cost model.
That matters most if a surprise bill has already burned you. A scraper that retries blindly after a timeout can pay twice for one page. A scraper that retries with a key, and an agent loop that knows when to stop, pays once.
TL;DR
- One key per logical operation (one URL, one extraction job, one tool call), never per HTTP attempt.
- On retry, resend the same key with the identical method, path, query and body. Change the request and you need a new key.
- Keys expire. Documented windows are commonly 24 hours, and some gateways use minutes. Never rely on a key outside the window.
- A replay is not always free: some providers still consume credits on replayed responses, others do not. Read the docs before you budget.
- Idempotency does not replace backoff, retry limits or spending caps. It works alongside them.
Table of Contents
- What an idempotency key does for a scraper
- Generating and reusing keys
- What the server should do with duplicates
- Key lifetime and what a replay costs
- What idempotency keys do not give you
- When to stop retrying
- Idempotent scraping with agents and MCP
- Where Gyrence fits
- Our view
- FAQ
- Sources
What an idempotency key does for a scraper
Most scraping calls look read-only, so it is tempting to assume retries are harmless. They are not, for two reasons. The call may start a job (a crawl, an extraction, a monitor) that has side effects. And the call may be metered, so each execution costs credits.
The AWS Builders' Library states that APIs with side effects are not safe to retry unless they provide idempotency, meaning the side effects happen only once no matter how many times you retry. It also notes that read-only APIs are typically idempotent while resource-creation APIs may not be.
The mechanism is simple. Per MDN's documentation of the Idempotency-Key header, the client sends a unique key with a POST or PATCH request. For a new key the server does the work and stores the key. For a repeat it skips the work and responds as though it had done it. A client that never received an acknowledgment can therefore resend safely.
Google Cloud's guide to idempotency puts the same idea in one line: repeated applications produce the same final effect with no extra side effects. That is the property you want when a network drops the response but the server already did the work.

Generating and reusing keys
The common failure is generating a new key on every attempt. That defeats the whole mechanism, because the server sees each retry as a new operation.
The rule, as Stitch's idempotency docs describe it: generate a new key for each logical operation, store it with the operation, and retry an ambiguous request with the same key and the exact same method, path, query parameters and body. Generate a new key only for a new logical operation. Stitch recommends UUIDs created once and persisted with the operation.
For scraping, define "logical operation" deliberately:
| Operation | Key scope | New key when |
|---|---|---|
| Fetch one URL for a snapshot | URL + snapshot run ID | The next scheduled run starts |
| Structured extraction | URL + schema version + run ID | You change the schema or prompt |
| Site crawl | Start URL + depth/limit settings + run ID | You change the crawl settings |
| Agent tool call | Agent task ID + step number | The agent deliberately re-asks |
The run ID in each row is what separates "retry the same job" from "collect again tomorrow." Without it, a key tied only to the URL would replay yesterday's page when you wanted today's.
Persist the key before you send the request. If your worker crashes after sending but before saving, you lose the key and the ability to retry safely. Write the key to your job table first, then call the API.
What the server should do with duplicates
Whether you are consuming a provider's implementation or building a thin layer of your own, the same cases come up.
Same key, same request, finished. Return the stored status and body. CaptainDNS documents a replay header, X-Idempotent-Replay: true, so the client can tell a replay from a fresh result.
Same key, request still running. MDN notes a server may return 409 while the original is still processing. Your client should wait and retry with backoff, not generate a new key.
Same key, different payload. MDN describes an "idempotency fingerprint," a hash of all or part of the request stored with the key, and a server may return 422 if the fingerprint differs. Clemta's docs bind the key to method, path, body and API version and refuse a mismatch. Treat this error as a bug in your code: you reused a key for something new.
Server error with unknown outcome. Clemta treats a 5xx as not a settled outcome and releases the key so a retry runs again, while replaying both successes and rejections. The MCP Runtime gateway takes the opposite, cautious position for tool calls: per its design record, it never re-executes a tool after an ambiguous 5xx. It stores a failed-unknown marker and returns 409 idempotency_outcome_unknown on retry. Know which behavior your provider has, because it decides whether your retry might run the job twice.

Key lifetime and what a replay costs
A key is not forever. Stitch states keys expire 24 hours after the first request, and Clemta remembers keys for 24 hours. The MCP Runtime gateway uses a 5-minute window and scopes keys to the verified principal, session and server. Retrying after the window means the server may treat your request as new.
Then there is billing, where providers genuinely differ:
- Talonic states a duplicate inside the window returns the cached response with no additional credit cost.
- CaptainDNS states that replays skip processing but credits are still consumed.
So "idempotent" does not automatically mean "free to retry." Put the replay-cost rule from your provider's docs in your cost model, then add a hard ceiling on top. A key protects you from double execution. It does not cap how many times your loop calls.
What idempotency keys do not give you
Two misconceptions cause most duplicate-data bugs.
A key is not content deduplication. Flexprice's glossary points out that keys do not prevent duplicate work when keys differ, and that protection ends at the provider's retention window. If two workers generate different keys for the same URL, you get two fetches.
A key does not make your pipeline exactly-once. The server-side guarantee covers the API call. Your own database writes and downstream alerts are separate. Put a unique constraint on your observation identity (normalized URL plus run ID plus parser version, for example) and write results with an upsert, so a replayed response lands on the same row. Any webhook or queue message you emit after the write needs its own deduplication on the consumer side.
When to stop retrying
Keys make retries safe. They do not make them wise. The AWS Well-Architected guidance (REL05-BP03) recommends exponential backoff, jitter to randomize intervals, and a limit on the maximum number of retries so the backend gets time to recover.
Apply that in a scraper with a simple decision table:
- Timeout or connection reset: retry with the same key, backoff plus jitter.
- Rate limit response: wait for the signaled delay, then retry with the same key.
- Permanent failure (not found, blocked by policy, schema mismatch): stop. Record the failure and move on.
- Retry budget exhausted: park the job in a dead-letter queue for inspection.
The branching only works if the response tells you which case you are in. That is why typed failures matter: a response that says what went wrong lets code decide, where an opaque error forces a blind retry. For the mechanics, see our posts on exponential backoff for scraping and scraping retries, and this walkthrough of scraper retry logic with jitter and a global deadline. For the concept in API-design terms, read idempotent web fetch explained. If your errors are the problem, scraping API errors covers the checks.
Idempotent scraping with agents and MCP
Agents retry on their own. A model that sees a timeout will often call the same tool again, sometimes with slightly reworded arguments. That is the worst case for keys: a reworded request is a different request.
Two habits help. First, have the orchestrator, not the model, own the key: derive it from the task ID and step, and inject it into the tool call. Second, return failures to the model as data it can reason about ("outcome unknown, do not re-run") rather than a raw exception. The MCP Runtime design above shows the principle: after an ambiguous failure, the safe answer to a retry is an explicit conflict, not a second execution.
Where Gyrence fits
Gyrence's Search, Traverse, Fetch, Extract and Map calls return typed, discriminated-union responses, failure cases included, so your retry logic can branch on what actually happened. Billing is credit-based with predictable cost per call and spending caps, and there is no separate AI extraction charge, so a retry loop has a ceiling and a known unit price. The Standard plan is $75 per month, Growth is $299 per month and Scale is $549 per month; Free, Founders and Pay-As-You-Go are also listed on the pricing page.
If you track pages for change, WebDoppler monitors them and sends webhook alerts. Webhooks are a downstream side effect, so deduplicate them on your consumer with your own key, as described above.
Our view
Retries are where scraping bills go wrong. Nobody plans to pay twice for a page, but a timeout, a loop and an agent with no memory will do it for you.
We build for that case. Every primitive returns a typed result, failures included, because a retry decision is only as good as the signal behind it. Spending caps exist so that when your logic is wrong, your bill is still bounded. Predictable cost per call means you can compute the worst case before you ship.
Our advice: key every logical operation, persist the key first, cap retries, and read the replay-cost rule of every API you depend on. Honesty about failure modes is the point of the design.
Glen
FAQ
Should I generate a new idempotency key on every retry?
No. A new key per attempt makes each retry look like a new operation, which is exactly the duplicate you were trying to avoid. Generate the key once per logical operation, store it, and reuse it with an identical request until you get a final answer.
Do idempotency keys make retries free?
Not necessarily. One provider documents that replays within 24 hours cost no extra credits, while another documents that credits are still consumed on replay. Check your provider's docs, and use a spending cap as the backstop.
What should my key be built from?
A random UUID persisted with the job works. If you want deterministic keys, derive them from the fields that define the observation: normalized URL, run ID, and schema or parser version. Leave out the attempt number, and include the run ID so tomorrow's collection is not mistaken for today's.
What happens if I reuse a key with a different request?
A well-built server rejects it. MDN describes a 422 response when the stored fingerprint differs, and some APIs return a named mismatch error. Fix it by issuing a new key for the changed request.
Sources
- Idempotency-Key header, MDN
- What is Idempotency? Google Cloud
- Timeouts, retries and backoff with jitter, AWS Builders' Library
- Idempotency, Stitch Docs
- CaptainDNS API Idempotency
- Idempotency, Clemta Docs
- Tool Call Idempotency ADR, MCP Runtime Docs
- Idempotency-Key Header, Talonic Docs
- Idempotency Key, Flexprice Glossary
- REL05-BP03 Control and limit retry calls, AWS Well-Architected

