Priority URL Scheduling: How Crawlers Decide What to Fetch Next

Priority URL scheduling is the part of a crawler that decides which known URL gets fetched next. Every discovered URL gets a score, the highest score leaves the queue first, and per-host limits decide whether it can go out right now. Get it wrong and you burn budget on low-value pages while the pages you care about go stale.

This guide covers the scoring signals, the two-level queue design, how adaptive scheduling differs from a fixed schedule, and how to handle new URLs, duplicates and failures.

Gyrence

Keep Crawl Spend Predictable

Gyrence’s web data API includes spending caps and site traversal, helping teams gather structured data without surprise scraping bills.

Explore the API

TL;DR

A priority scheduler scores every known URL and fetches the highest score first, subject to per-host rate limits.
Useful signals fall into three groups: importance, relevance to your purpose, and dynamicity (how often a page changes).
Adaptive scheduling learns each page's change rate and moves URLs between priority buckets. A fixed schedule treats every URL the same.
Politeness is a constraint on the scheduler, not an afterthought: back off on latency, 5xx and 429.
Cap spend and depth, so a bad priority function cannot become a bad invoice.

What priority URL scheduling actually is
Signals that belong in the priority score
Two-level scheduling: URLs and hosts
Adaptive scheduling versus a fixed schedule
Budgets, politeness and joint scheduling
New URLs, duplicates, failures and changed content
Seeding the frontier well
Our perspective
Run it on an API with caps built in
FAQ
Sources

What priority URL scheduling actually is

The classic framing comes from Cho, Garcia-Molina and Page. They treat URL ordering as selecting the next URL from the known set when storage, processing time and crawling resources are limited. Their ordering metric is a score computed only from what the crawler has already observed, and the highest-scoring URL is fetched next.

Two consequences follow:

The score can only use information you have. For a URL you have never fetched, that is the link text, the parent page, the anchor, the path, and the host's history.
The score is a guess. It should be cheap to compute and cheap to revise.

The data structure is usually called a URL frontier. It is not one queue. It is a priority structure plus a per-host throttle.

Signals that belong in the priority score

The Olston and Najork survey names importance, relevance to the crawl's purpose and dynamicity as the main dimensions in published crawl-ordering techniques. Practical versions:

Dimension	Example signals	Good for
Importance	Link count, PageRank-style scores, traffic, site reputation	Broad crawls, indexing
Relevance	Path patterns, anchor text, topic classifier on the parent page	Focused crawls, RAG ingestion
Dynamicity	Historical update rate, content-type (news vs. docs)	Monitoring, freshness
Cost	Depth, response size, host latency	Budget control

The same survey lists usefulness estimates built from PageRank, traffic, site reputation and historical update rate. Your crawl purpose decides the weights. A compliance team tracking filings cares about relevance and dynamicity, and ignores global popularity. A search index does the opposite.

Freshness versus age

A naive freshness metric asks whether your copy is identical to the live page. The Purdue lecture notes point out that freshness alone can produce poor decisions, and present page age as a more useful objective: the cost of leaving a page uncrawled grows the longer it sits. The same notes describe historical refresh rate as a priority heuristic, and application-specific rules such as visiting news sites more often.

Factors shaping page revisit priority

In practice: a page that changes hourly and a page that changed once in 2019 should not share a revisit interval, and a page you gave up on should not drop to zero priority forever.

Two-level scheduling: URLs and hosts

Ranking URLs globally is not enough, because you cannot hit one host at full speed. Tu's Princeton report uses two levels: URLs are ranked inside per-server queues, and the servers themselves are ranked in a server queue.

That gives you a clean loop:

Pick the host whose best URL has the highest priority and whose politeness timer has expired.
Pop that host's best URL.
Fetch, record outcome, rescore, and push new links back in.

There is a trade-off. The same report notes that batching several requests per connection improves network performance but reduces the granularity with which URL priorities take effect, since established connections are not preempted. If strict ordering matters more than throughput, keep batches small.

Adaptive scheduling versus a fixed schedule

A fixed schedule says "recrawl everything every 24 hours." It is simple, predictable and wasteful: stable pages are refetched for nothing, and fast-changing pages are visited too rarely.

Adaptive scheduling estimates each page's change dynamics from observation and changes the interval. Microsoft Research describes the problem as learning unknown page-change dynamics while optimizing freshness and information completeness under crawler and host bandwidth limits.

A patent filing from Google, US20130144858A1, shows one concrete mechanism. Popular resources get shorter minimum crawl intervals, and resources whose observed change rate is materially higher can be promoted to higher-priority buckets. You do not need the patent's full machinery. A three-bucket version works:

Hot: changed on the last two visits. Revisit soon.
Warm: changed on one of the last few. Revisit at the base interval.
Cold: unchanged for many visits. Back off the interval, with a ceiling so it never stops.

For the math behind refresh policies, Cho and Garcia-Molina's ACM study models page updates as a Poisson process and compares policies using two freshness metrics. It is a good starting point if you want to evaluate your own scheduler instead of trusting intuition.

Budgets, politeness and joint scheduling

Priority is only half the problem. The other half is what the host will tolerate. Google's crawl budget documentation defines a site's crawl budget as the URLs the crawler can and wants to crawl, combining capacity and demand. It also says capacity drops when latency rises or when 5xx or 429 responses show up. Capacity itself is about limiting total connection-holding time across parallel connections.

Copy that behavior:

Track per-host p50 latency and error rate.
On a 429 or a run of 5xx, widen that host's delay and lower its server-queue rank.
Prefer honoring any Retry-After header over a fixed guess.

Research formalizes this. The SIGIR 2019 paper frames freshness scheduling around page importance, change rates, a crawler-wide request budget and per-host politeness limits. Its authors report PoliteBinaryLambdaCrawl as an optimal algorithm under those constraints. For the site-owner side of the same trade-off, this walkthrough on reclaiming crawl budget on large sites shows what inefficient URL inventory costs when a crawler has a fixed allowance.

We cover the practical defaults in polite crawling and the backoff side in scraping rate limits.

New URLs, duplicates, failures and changed content

Four cases cause most scheduler bugs. The crawler-commons URL frontier API gives a clean model for them:

Newly discovered URL: add only if not already known, and serve it as soon as possible.
Known URL: schedule a refetch after a specified date, rather than re-adding it.
Transient error: retry later with a new date, not immediately.
Duplicate: drop before scoring. Normalize first (case, fragments, tracking parameters, trailing slashes). Our piece on URL deduplication shows how much bandwidth this saves.

Distinguish failure types in your state, not just "failed." A 404, a timeout, a robots block and a 429 all deserve different next actions. Hiding that distinction is how crawlers silently decay.

Seeding the frontier well

What you put in first shapes everything after. Sitemaps give you a URL list with hints, but they can be stale or capped; see what to do when sitemaps hit 50,000 URLs and the sitemap discovery audit. Then bound exploration with crawl depth limits, since depth is a cheap proxy for diminishing value.

Our perspective

Most scheduling write-ups treat the scheduler as a pure ranking problem. In production it is a spending problem. A priority function that is slightly wrong costs you pages. A priority function that is slightly wrong and uncapped costs you an invoice.

So we build around three rules:

Every call has a typed outcome. Success, blocked, rate-limited, empty and failed are different variants of a discriminated union, so a scheduler can branch on them instead of parsing error strings.
Cost is bounded before ranking starts. Spending caps and depth limits sit outside your priority logic, so a bug in it cannot run away.
Rank on evidence. Use observed change and observed failure, not assumptions about what the web should do.

We would rather tell you a fetch failed and why than return a confident-looking empty page. Web data is a storm, and a scheduler is only as trustworthy as the signals it is fed.

— Glen

Run it on an API with caps built in

If you would rather not build the frontier, host throttles and retry logic yourself, Gyrence gives you Search, Traverse, Fetch, Extract and Map as composable primitives, with typed failure responses and spending caps. Use Map to enumerate a domain's URLs, then decide in your own code which ones deserve a Fetch or Extract call. Plans are listed on the pricing page, including Standard at $75 per month. For change detection on pages you already care about, see WebDoppler.

Start on the free tier, set a cap, and see what your crawl really costs before it scales.

FAQ

What is priority scheduling in a web crawler?

It is an ordering policy for the crawl queue. Each known URL gets a score from signals such as importance, relevance and change history, and the highest score is fetched next. Per-host politeness limits then decide whether it can go out immediately.

How is adaptive crawl scheduling different from a fixed schedule?

A fixed schedule revisits every URL at the same interval. An adaptive one observes how often each page changes and shortens or lengthens its interval accordingly, so fast-changing pages are visited more and stable ones less.

Which signals should I weight first?

Start with relevance to your purpose and observed change rate, because you can measure both yourself. Add importance signals such as inbound links only if you run a broad crawl. Add cost signals such as depth and host latency to protect your budget.

How do I stop a scheduler from hammering one site?

Keep a per-host timer and a per-host queue, and rank hosts as well as URLs. Back off when latency climbs or when you see 5xx and 429 responses, which is also how Google describes lowering its own capacity for a site.