A content inventory crawl is a controlled traversal of a site that records one row per page or asset: URL, title, type, dates, status and whatever metadata your audit needs. Do it in four moves. Define the scope, map the URLs, fetch each page, then extract the same fields from every page. The rest of this guide covers each move and the failure modes that quietly corrupt an inventory.
TL;DR
- An inventory is quantitative: a list of what exists. An audit is qualitative: a judgment on what to keep, fix or remove.
- Write down scope and exclusions before crawling: subdomains, file types, paths, protocols.
- Seed the crawl from a URL map, then traverse to catch pages the sitemap misses.
- Record failures as rows (404, redirect, blocked, timeout) instead of dropping them.
- Cap spend and depth up front so a large site cannot surprise your bill.
Table of Contents
- Inventory first, audit second
- Define scope before you crawl
- Which fields to collect per page
- Map first, then traverse
- Handle errors, duplicates and missing metadata
- From inventory to Markdown, JSON and RAG
- Check pricing and limits before the big crawl
- Perspective
- Run the inventory with Gyrence
- FAQ
- Sources
Inventory first, audit second
The two terms get mixed up. Nielsen Norman Group describes an inventory as a list of content at the page or asset level, while the audit evaluates quality and identifies what needs updating or removal. The U.S. government's Digital.gov guidance says much the same: an inventory gives you a comprehensive list of a site's documents and is a good way to start an audit.
That split matters for how you build the crawl. The crawl's job is mechanical and repeatable. It should produce the same rows tomorrow that it produced today, unless the site changed. Judgment comes later, on top of clean data. If you want a walkthrough of the evaluation half, this guide on how to perform a content audit picks up where the crawl ends.
Define scope before you crawl
Most bad inventories are scope problems, not tooling problems. NN/G recommends defining scope systematically: which sites, domains, subdomains and external resources are in, and which exclusions apply, such as JavaScript files, CSS files, extensions, paths and protocols. Decide these questions in writing:
- Whole site, a section, or a user journey? Digital.gov notes you can start from the homepage and trace pages through user task flows. That is a valid inventory when the goal is a single journey, such as checkout help.
- Subdomains. Is
blog.ordocs.part of the same content estate? If yes, list them explicitly. - Assets. PDFs, images and downloads count as content for many audits. Decide whether you want them as rows or only as links.
- Exclusions. Faceted URLs, session parameters, print views and staging paths inflate the count without adding content.
A goal sentence helps: "Find every page older than two years that still gets linked from navigation." That sentence tells you which fields to collect.
Which fields to collect per page
Keep the schema small enough that every row can be filled. A technical planning guide from Content Aspects lists titles, dates, language, source URLs, aliases and content type as typical inventory fields. A practical baseline:
| Field | Why it earns its column |
|---|---|
url (requested) and final_url | Shows redirects and aliases |
canonical_url | Exposes duplicates |
http_status | Separates live pages from dead ones |
content_type | HTML, PDF, image, feed |
title, h1 | Quick human review |
published, modified | Staleness checks |
language | Multi-region sites |
word_count | Thin-content screening |
depth and parent_url | Where the page sits in the link graph |
fetched_at | Source traceability |
The last row is the one teams skip and regret. For compliance and financial research, an inventory row without a fetch timestamp and source URL cannot be traced back to what the page said at the time.
Map first, then traverse
Two discovery methods find different things. A sitemap lists what the site owner intends to publish. A link traversal finds what is actually reachable. The gap between them is itself a finding: orphaned pages appear in the sitemap but not in navigation, and unlisted pages appear in navigation but not in the sitemap.
Our approach with the Gyrence API follows that order:
- Map the domain to get the URL graph from sitemaps. We cover the method in site inventory via URL mapping, and what to do when a sitemap grows past its limits in this sitemap versus crawl guide.
- Traverse outward from the homepage or a section root to find linked pages the map did not list.
- Diff the two sets. Anything in only one set gets flagged for review.
- Fetch and extract the fields from the union.
Set crawl depth deliberately. Unbounded depth on a site with calendar pages or search results produces thousands of near-empty URLs. Our notes on safe crawl depth defaults explain how to choose limits, and polite crawling defaults cover rate limits and 429 responses.
Handle errors, duplicates and missing metadata
The inventory is only as honest as its handling of things that go wrong. The Content Aspects guide suggests checking for 404s, resolving case-sensitivity issues, de-duplicating results and forcing redirections during execution. Turn that into rules:
- Keep failures as rows. A 404 linked from your navigation is a finding. A timeout is a retry candidate. A robots-blocked page is a scope decision. If your tool silently drops them, your count is wrong and you will not know by how much.
- Normalize before de-duplicating. Lowercase hosts, strip tracking parameters, resolve trailing slashes, then compare canonical URLs. See our five validation steps for canonical URLs.
- Mark missing metadata as null, not empty string. A missing
modifieddate and a date that is literally blank mean different things. - Track redirects. If you are inventorying ahead of a relaunch, the redirect map is the deliverable. A one-to-one redirect checklist shows how the inventory feeds a migration plan.
This is where typed responses pay off. When every call returns a discriminated union (success, redirect, blocked, timeout, unsupported type), your pipeline can branch on the result instead of parsing error strings or guessing why a field is empty.
From inventory to Markdown, JSON and RAG
An inventory row tells you a page exists. Many teams then want the content itself, for an audit spreadsheet, a migration, or a retrieval index for an AI agent. Two outputs cover most needs:
- Clean Markdown for reading, diffing and chunking. Our guide to fetching web pages to Markdown for LLMs covers what gets stripped and what stays.
- Structured JSON from an extraction prompt or schema, when you need typed fields such as author, product count or last-reviewed date.
Non-HTML content needs its own step. A U.S. federal statistical-methods paper on web crawling and text tools notes that PDF documents may need conversion to a format more amenable to analysis. Route by content type so a PDF is never treated as a broken HTML page.
For RAG, add validation before anything reaches the vector store: minimum content length, boilerplate detection, and a stored source URL with fetch time for every chunk. Crawling and scraping are different cost shapes, which we break down in crawling versus scraping for RAG.
Check pricing and limits before the big crawl
Inventory crawls are bursty. A site you guessed at 2,000 pages turns out to have 40,000. Before you run, ask:
- Is the unit of billing a call, a credit, a page or a compute minute?
- Is there a hard spending cap, or only an alert after the fact?
- Is LLM extraction billed separately from fetching?
- What happens when a page fails: do you pay for it, and do you learn why?
- Can you cap depth and total pages per run?
On our side, Gyrence bills in credits with spending caps, and there is no separate AI extraction charge. Current plan details are on the pricing page.
Perspective
Most content inventories fail quietly. The crawler hits a wall of 429s, drops a few thousand URLs, and reports a tidy total that looks authoritative. Nobody finds out until the audit recommends deleting a page that was never actually checked.
We built Gyrence to refuse that kind of silence. Search, Traverse, Fetch, Extract and Map each return a typed response, and the failure cases are first-class values, not exceptions buried in logs. For an inventory, that means a blocked page is a row with a reason, and your totals mean what they say.
The other half is cost. A crawl you cannot cap is a crawl you cannot schedule. Spending caps and predictable credit billing let you run the inventory monthly and diff it, which is when it becomes useful.
Glen
Run the inventory with Gyrence
If you would rather not stitch together a mapper, a crawler, a Markdown converter and an extractor, Gyrence exposes all five primitives through one API and a hosted MCP endpoint, so an agent can run the same workflow. Paid plans are Standard at $75 per month, Growth at $299 per month and Scale at $549 per month, alongside Free, Founders and Pay-As-You-Go options.
Start on the pricing page, or set up WebDoppler to get webhook alerts when pages on your inventory change after the first crawl.
FAQ
What is the difference between a content inventory and a content audit?
An inventory lists what content exists, page by page or asset by asset. An audit evaluates that content for quality and decides what to update, merge or remove. Run the inventory first, because the audit needs its data.
How often should I re-crawl for a content inventory?
It depends on how fast the site changes. A quarterly crawl suits most marketing sites, while a docs or catalog site may need weekly runs. Change monitoring between full crawls keeps the inventory current without re-fetching everything.
Should the inventory include PDFs and images?
Include them if users reach them from search or navigation, which is true for most sites. Record them as rows with their content type and source page. Convert PDFs to text only when you need to analyze their content.
Is a sitemap enough to build an inventory?
No. A sitemap shows what the owner listed, not what is linked or live. Combine it with a link traversal and compare the two sets. The differences are usually where the problems are.
Sources
- Content Inventory and Auditing 101, Nielsen Norman Group
- Content goals, Digital.gov
- Planning Your Content Inventory: A Technical Quick Guide, Content Aspects
- SABLE: Tools for Web Crawling, Web Scraping, and Text Analysis, FCSM 2018
- How to Perform a Content Audit for Increased Organic Traffic
- Preserve Rankings with One-to-One Redirects and a 30-90 Day Watch Plan