← Back to blog

Sitemap Index Crawling: How to Fetch Every Child Sitemap

November 13, 2026
Sitemap Index Crawling: How to Fetch Every Child Sitemap

Sitemap index crawling means fetching a sitemap index, reading each child sitemap it lists, and only then collecting page URLs. A crawler that treats the index as a flat URL list returns zero pages. A crawler that swallows errors returns a partial list and calls it complete. The reliable version discovers the index, follows children to a bounded depth, handles gzip, deduplicates, and reports every failure as data.

Gyrence
Turn Site Structure Into Usable Data
Explore Gyrence's web data API for searching, traversing sites, fetching pages, and mapping domain URL graphs with typed responses.
Visit

TL;DR

  • A sitemap index lists other sitemaps, not pages. Each Google sitemap index can hold up to 50,000 child sitemaps, and each child is capped at 50 MB uncompressed or 50,000 URLs.
  • Discover from robots.txt Sitemap: lines first, then fall back to conventional paths such as /sitemap.xml and /sitemap_index.xml.
  • Detect gzip by magic bytes, cap depth, and deduplicate with a visited set.
  • A sitemap is a hint. Submitting or listing one does not guarantee that a search engine, or your crawler, gets every URL.
  • Record per-child status (fetched, empty, blocked, malformed) so a partial read is visible rather than silent.

Table of Contents

What a sitemap index is

Google's documentation says a sitemap index file is used when a sitemap exceeds size limits, so many sitemaps can be submitted at once. It uses the standard sitemap XML namespace, a sitemapindex root tag, a sitemap parent tag and a loc child tag, with up to 50,000 loc tags.

The limits for each child come from Google's build and submit guidance: 50 MB uncompressed or 50,000 URLs per sitemap, UTF-8 encoding. A site with a million pages therefore needs at least 20 child sitemaps, and large sites often split further by language, section or content type.

The tree can be deeper than one level. The ultimate-sitemap-parser documentation models a sitemap tree where index sitemaps list other sitemaps, which may themselves be indexes, while page sitemaps list pages. It even treats robots.txt as an index-type node, since it points to sitemaps. Your crawler needs to expect several levels before it reaches a page.

Nested sitemap indexes leading to page sitemaps

How discovery works

Start where the site owner declares intent, then fall back to convention.

  1. Read robots.txt. Collect every Sitemap: directive. There can be several, and they can point to indexes or to page sitemaps.
  2. Probe conventional paths. Try /sitemap.xml and /sitemap_index.xml. The AWS Prescriptive Guidance crawler example checks several locations, including /sitemap_index.xml and /sitemap/sitemap.xml, verifies the response is XML, and recursively fetches sub-sitemaps when it meets an index.
  3. Fall back to other signals. The same guidance lists HTML sitemaps, crawling from the homepage, URL pattern analysis and the site: operator for sites with no usable sitemap.

The DuckDB sitemap extension shows how far discovery can go: robots.txt, standard paths, HTML meta tags, a brute-force finder over 587+ URL patterns, recursive traversal with a default max_depth of 3, exponential backoff retries, and respect for Retry-After on 429 responses. For a deeper walk through discovery, see our 15-minute sitemap discovery audit.

The mistakes that break index crawlers

Treating an index as a urlset

Per the invisible_playwright wiki, the most common way sitemap scraping breaks is iterating a document as a urlset when it is really an index. Branch on the root element: sitemapindex means recurse, urlset means collect.

Trusting the file extension for gzip

Child sitemaps are frequently served as .xml.gz, because an uncompressed urlset for a large site runs to many megabytes. The master-fetch library goes further and detects gzip by magic bytes (\x1f\x8b) instead of the extension, which covers servers that compress without renaming. Check the first two bytes, then decompress.

Unbounded recursion and duplicates

Indexes can reference each other, and the same URL often appears in several child sitemaps. Use a visited set for both sitemap URLs and page URLs. Set explicit ceilings. The same library defaults to 3 index levels, 25 sitemaps and 5,000 URLs, a sensible pattern even if you raise the numbers. Our guide to crawl depth limits covers safe defaults.

Believing lastmod blindly

One SEO reference notes that Google only trusts lastmod when it is consistently and verifiably correct, and that a CMS resetting the date site-wide on every rebuild is the biggest mistake. For your own crawler, that means lastmod is a scheduling hint. Compare it with content hashes before you decide a page changed. The same page says Google ignores changefreq and priority.

A sitemap is a hint, not a guarantee

Google states that submitting a sitemap is only a hint and does not guarantee it will download or use the file. Its overview of sitemaps adds that a sitemap helps discovery but does not guarantee every item is crawled and indexed, and that it matters most for large sites, new sites with few external links, and sites with rich media.

The practical consequence for a data pipeline is that the sitemap gives you candidates, not coverage. Discovery guidance on dynamic sitemaps for developers makes the same distinction between supporting URL discovery and promising indexing. Treat a sitemap as one input alongside link traversal, and reconcile the two. If you want the trade-offs, read what happens when sitemaps hit 50,000 URLs.

Why some child sitemaps never get read

Partial reads are the hard failure, because the index itself looks healthy. One engineer documented a case where Search Console reported success on an index while every child sitemap was invisible to Google. The Sitemaps API returned isPending: true, zero errors and no download timestamp, meaning no fetch of the children had been attempted. The cause was a WAF per-IP rate limit that challenged bursts of 63 child fetches from one Google IP within seconds, putting those URLs into a per-URL failure backoff. Resubmitting the file at a versioned URL bypassed the backoff.

The lesson applies to your crawler too. Firing every child fetch at once from one IP looks like an attack. Pace requests, honor 429 and Retry-After, and log which children were challenged. Our notes on polite crawling defaults and scraping rate limits cover the numbers.

A second cause is visibility. For sitemaps of sitemaps, Google's John Mueller explained that Search Console does not show page-level details directly, so submitting child sitemaps individually may be needed to see them. Also confirm that robots.txt and robot meta rules allow what you fetch. A single server error on robots.txt can change your crawler's behavior, as covered in robots.txt compliance and robots meta tags.

Why some child sitemaps never get read

What to record so results stay traceable

Silent failure is the real enemy. The master-fetch library never raises: a failed fetch or parse returns an empty result and the crawl falls back to breadth-first search. That is convenient, but an empty list is indistinguishable from an empty site. For agent and RAG workflows, store a record per node in the tree:

FieldWhy it matters
Sitemap URL and parent URLRebuild the tree and trace any page back to its index
HTTP status and Retry-AfterSeparate blocked from missing
Compression detectedExplain parse failures
Depth and URL countShow limits were hit, not that the site ended
lastmod as declared, plus your own content hashAudit freshness claims
Outcome: ok, empty, blocked, malformed, cappedMake partial reads explicit

That outcome field is the point. An agent that receives capped or blocked can retry, narrow scope or tell the user. An agent that receives [] guesses. This is a design position, not a standard: the research I checked on this question comes from library authors and engineers, and no spec mandates such a log.

For retrieval pipelines, one engineer's sitemap-to-RAG architecture crawls production-rendered HTML, cleans and chunks it, and emits a versioned artifact. The author warns against crawling recursively and accepting every link, because navigation exposes query strings, faceted archives, previews and external URLs, and recommends a manifest so inclusion is intentional. A sitemap is a natural starting manifest, filtered by your own rules.

What we think

Most crawler bugs we see described around sitemaps are not parsing bugs. They are honesty bugs: a function returns an empty list when it means "blocked" or "too deep." We built Gyrence's Map primitive, which maps a domain's URL graph through sitemaps, on the opposite rule: every call returns a typed, discriminated-union response, including the failure cases, so your code or your agent can branch on what actually happened.

The same applies to cost. A sitemap index with hundreds of children can turn into hundreds of fetches. Spending caps and predictable cost per call mean the bill is something you set, not something you discover. Pair Map with Traverse to reconcile what the sitemap claims against what the links show.

The sitemap tells you what the owner wants found. The traversal tells you what is there. You want both, and you want the gaps reported.

Glen

Run it without building the plumbing

Gyrence exposes Search, Traverse, Fetch, Extract and Map as one API, plus a hosted MCP endpoint for agent clients. Map handles URL discovery through sitemaps, Fetch cleans a page to markdown, and Extract returns schema-guided JSON with no separate AI extraction charge. Every response is typed, failures included. WebDoppler adds monitoring with webhook alerts.

Plans on the pricing page include Free, Founders and Pay-As-You-Go, plus Standard at $75 per month, Growth at $299 per month and Scale at $549 per month.

Gyrence

FAQ

Can a sitemap index point to another sitemap index?

The tree model used by sitemap parsers allows index sitemaps to list other indexes, so crawlers should handle several levels. Many tools cap depth, with 3 as a common default, to avoid loops and runaway fetching. Set your own limit and record when you hit it.

Why does my crawler return zero URLs from a valid sitemap?

Usually it is reading a sitemapindex as if it were a urlset, or it fails to decompress a .xml.gz child. Branch on the root element and detect gzip by the first two bytes. Also check whether a WAF is challenging your requests.

Does listing a URL in a sitemap guarantee it gets crawled?

No. Google says a sitemap helps discovery but does not guarantee that every item is crawled and indexed. Your own crawler should likewise treat sitemap URLs as candidates and track which ones were actually fetched.

What are the size limits for a sitemap index?

Per Google, an index can list up to 50,000 sitemaps, and each child sitemap is limited to 50 MB uncompressed or 50,000 URLs. Files must be UTF-8 encoded, and compression is allowed.

Sources