Queue-based crawling is a design where a crawler keeps its pending URLs in a queue, pulls one at a time, fetches it, extracts new links and pushes eligible ones back. The queue is the crawl's state. Get it right and the crawl survives crashes, respects each host and never fetches the same page twice. Get it wrong and you get duplicates, lost pages and angry site owners.
TL;DR
- A crawler seeds a queue, takes a URL, downloads the page, extracts links and enqueues new ones until a stop condition is met (academic overview).
- A plain global FIFO is not enough. You need per-host scheduling, URL canonicalization and priorities.
- Durable state lets workers crash and be replaced without stopping the crawl.
- Retries must never lose a URL, and failures should be recorded as typed outcomes, not swallowed.
- Cost and scope limits belong in the queue layer, not bolted on afterward.
Table of Contents
- What the queue actually does
- Why a queue beats an in-process loop
- Frontier design: store first, schedule second
- Politeness: one queue per host
- Retries without losing URLs
- Recrawling: freshness versus cost
- Making crawled data usable for RAG and agents
- Perspective
- Skip the infrastructure: Gyrence
- FAQ
- Sources
What the queue actually does
The loop is short. Start with seed URLs. Take one from the queue. Fetch it. Parse the response for links. Filter them against scope rules. Add the survivors. Repeat until the queue is empty or a limit trips.
The Internet Archive's crawler splits this into named parts. Its Frontier hands out the next URI, maintains politeness and tracks which URIs are discovered, in process or done. Worker threads pull from it and run processors in order: pre-fetch checks (DNS, robots), fetch, link extraction, then writing. Discovered links are filtered and scheduled back into the Frontier.
That separation is the pattern to copy: queue, fetcher, parser and storage as distinct components with a clear contract between them.
Why a queue beats an in-process loop
In a simple async crawler, state lives in process memory and dies with the process. In a queue-based crawler, state lives in the broker, so a worker is disposable. It can crash, be replaced or be joined by more workers and the crawl keeps going.
The same source recommends a few practical defaults:
- Separate queues for cheap HTTP fetches and expensive browser renders.
- Async I/O inside each worker.
- A capped frontier (
maxmemorywithnoevictionin Redis) so it fails loudly instead of silently dropping URLs. - At-least-once delivery, which means tasks must be idempotent.
Frontier design: store first, schedule second
A frontier entry needs more than a URL. One design keeps the URL, its origin, priority, next eligible time and attempt state. A scheduler picks among origins whose next fetch is due and leases a bounded batch to a worker.
Persist every discovered URL in a state store first, then promote eligible ones into the work queue. That keeps discovery separate from dispatch and makes the crawl inspectable.
Canonicalization and deduplication
Two different jobs get confused here. URL canonicalization removes redundant address variants. Document deduplication stops identical content from entering storage and indexing twice. System Design Space stresses keeping them separate.
For canonicalization, one open-source crawler normalizes URLs per RFC 3986 before using them as dedup keys: lowercase scheme and host, drop default ports, strip trailing slashes, remove empty query strings and strip fragments. Normalizing too aggressively can merge distinct pages, so see our guide on validating canonical URLs before you ship rules.
Priorities
A frontier typically issues URLs by priority, crawl depth and fairness across hosts. Common priority signals are depth from the seed, sitemap membership, page type and time since last fetch. We cover the trade-offs in priority URL scheduling and in crawl depth limits.
Politeness: one queue per host
The classic mistake is a single global FIFO. It hammers one host while others wait. The fix is a queue of hosts, each with its own queue of URLs. Jim Mischel's write-up shows the effect: with 100 domains, you visit the first once, then the other 99 before returning to it. Politeness comes for free and no worker sits idle waiting.
The scheduler should check each host's budget and honor any crawl-delay before dispatch. System Design Space lists the usual errors: global FIFO without per-host limits, no canonicalization, the same recrawl cadence for everything, queues held only in RAM and ignoring robots.txt.
Practical defaults for the host layer:
- Track
next_eligible_atper host. - On a 429, back off and widen the delay. See polite crawling defaults and rate-limit handling.
- Read robots.txt before the first fetch, and treat server errors on it carefully: one 5xx can block a site.
- Read page-level directives too, covered in robots meta tags.
Robots.txt is advisory, not a shield
Robots.txt is a signal, not an access-control mechanism. One legal analysis notes that courts have generally declined to treat robots.txt violations alone as CFAA breaches, though contract and tort theories may still apply. Build compliance into the queue anyway. For audit-ready controls, see our compliance-friendly scraping checklist. If you need legal advice for your use case, talk to counsel.
Retries without losing URLs
A retry must not drop the URL. The reliable pattern: mark the entry in flight with an expiry. If the worker dies, the lease lapses and the URL becomes eligible again. If a worker stores the body but crashes before reporting completion, a content digest plus an idempotent result record prevents duplicate artifacts.
Record a typed outcome for every attempt: success, blocked, rate-limited, timeout, content-type mismatch. A queue that only knows "done" or "not done" cannot tell you why a page is missing.
Recrawling: freshness versus cost
Recrawling everything at one cadence wastes bandwidth. Recrawling is a constant balance between freshness and network cost. Give volatile pages short intervals and stable ones long ones, and use change signals such as ETags and content hashes. The same logic drives crawl budget audits on large sites, where deciding when to revisit pages is the central question.
Making crawled data usable for RAG and agents
A queue gets pages in. Retrieval quality depends on what you keep. One provenance guide suggests capturing immutable fetch evidence with checksums, validating before parsing, normalizing deterministically, gating on quality and drift, and indexing with source and lifecycle fields. Store URL, fetch time, status and content hash with every chunk so answers can be traced back. For a deeper look at the crawl-versus-extract split, read crawling vs. scraping for RAG.
Perspective
Most crawlers fail quietly. A URL vanishes, a 429 becomes an empty string, and the pipeline reports success. We built Gyrence around the opposite rule: every call returns a typed, discriminated-union response, failures included, so an agent can reason about what happened instead of guessing.
That is why Traverse, our site-outward crawl primitive, sits next to Fetch, Extract and Map rather than hiding the frontier behind a black box. You decide scope and depth, and spending caps keep a runaway crawl from becoming a surprise bill. Running your own queue is a legitimate choice. If you would rather not operate brokers, leases and per-host schedulers, the same principles are available as an API.
Glen
Skip the infrastructure: Gyrence
Gyrence handles frontier, politeness and typed failures behind five primitives (Search, Traverse, Fetch, Extract, Map) and a hosted MCP endpoint. Plans are Free, Founders, Pay-As-You-Go, Standard at $75 per month, Growth at $299 per month and Scale at $549 per month. Extraction is bundled with no separate AI charge. Details are on the pricing page.
If you need change alerts on pages you already track, look at WebDoppler. For crawling many domains at once, see bulk domain crawling.
FAQ
Should the crawl queue be FIFO or a priority queue?
Use a priority queue, partitioned by host. Pure FIFO ignores depth, importance and fairness. Priority within each host queue, plus round-robin across hosts, gives you both useful ordering and politeness.
Where should the queue live?
In a durable broker or database, not only in process memory. Memory-only queues lose all state when the process dies. If you use Redis, cap memory and disable eviction so overflow is an error, not silent data loss.
How do I avoid crawling the same page twice?
Canonicalize URLs before they become dedup keys, then deduplicate content with a digest after the fetch. The first removes address variants, the second catches different URLs serving the same document.
Does queue-based crawling replace scraping?
No. Crawling discovers and fetches pages. Extraction turns each page into structured fields. Most pipelines run both, with the queue feeding the extractor.
Sources
- Overview of a web crawler application (academic paper)
- Internet Archive crawler developer manual: overview
- Distributed crawling with Celery and Redis
- ScaleWiki: web crawler politeness, scope and reliable progress
- System Design Space: web crawler
- Spatie crawler v9: custom crawl queue
- Writing a web crawler: queue management, part 1
- Data scraping after hiQ v. LinkedIn
- Source ingestion and provenance pipeline
- Crawl budget optimization audit