Visit Website →
← Back to blog

Crawling vs. Scraping Keeps RAG Costs Capped with Typed JSON

October 18, 2026
Crawling vs. Scraping Keeps RAG Costs Capped with Typed JSON

Crawling discovers and indexes pages; scraping extracts structured data from those pages. Crawlers follow the rules in RFC 9309 and the guidance in Google Search Central, while scrapers turn pages into JSON records. Below, we cover how each works, where they differ, and which controls keep your pipeline polite and predictable.

TL;DR:

  • Use breadth first for unknown sites, depth first for a specific section, and set per host page and depth caps to control crawl spend.
  • Fetch pages with plain HTTP first; inspect for target fields, then use headless rendering only when JavaScript hides the needed content.
  • Treat robots.txt as a crawling request, not permission to reuse content; commercial use may require authorization, especially for licensed material.
  • Validate extracted records against a schema, normalize values at ingestion, and retain each source URL and fetch timestamp to trace errors and assess freshness.
Gyrence
Keep Web Data Costs Predictable
Gyrence combines site traversal and structured extraction with spending caps and typed responses for agent-ready web data.
Explore Gyrence

Table of Contents

What is web crawling?

Web crawling is the automated, recursive traversal of links to discover pages and build a map of a site's URLs. The Robots Exclusion Protocol, specified in RFC 9309, tells a crawler which paths a site asks it to avoid, and it is the current technical standard for parsing those rules.

A working crawler is built from a few parts:

  • Seed URL: the starting point, such as a homepage or a sitemap.
  • Queue: a frontier of URLs waiting to be visited, deduplicated so the same page is never fetched twice.
  • Link parsing: extracting and normalizing every link found on each fetched page.
  • Scheduling and caching: deciding when to revisit a URL and when a cached copy is good enough.

Crawlers power search indexes, site maps, and large-scale discovery jobs. Their output is a graph, not a dataset. Here is what a small run might return:

/                      -> successful response, links: /pricing, /blog, /docs
/blog                  -> successful response, links: /blog/post-1, /blog/post-2
/docs/old-page         -> not-found response

Nothing in that output tells you a price or a title. It tells you what exists and how it connects.

What is web scraping?

Web scraping is focused extraction of specific fields or records from a page into a structured format such as JSON or CSV. Where a crawler asks "what pages exist?", a scraper asks "what is the value on this page?"

Common techniques include:

  • HTML parsing with selectors: CSS or XPath selectors pull the exact element you want.
  • HTTP-first fetching: request the page directly and parse the response, which is fast and cheap.
  • Headless rendering: run a real browser engine when the content only appears after JavaScript executes.
  • Schema-guided extraction: describe the fields you want and let an LLM fill a typed JSON object, covered in our structured JSON extraction guide.

Downstream, scraped data feeds price trackers, RAG sources, monitoring alerts, and analytics. A concrete contrast: scraping a product page returns {"name": "Trail Shoe", "price": "example price", "currency": "USD"}. Crawling the same merchant returns thousands of product URLs and no prices at all.

Key differences: scope, intent, output, and scale

The cleanest way to decide what you need is to compare four axes.

  • Scope: a crawler works across a whole site or the open web. A scraper works on one page or a known set of pages.
  • Intent: crawling serves discovery and indexing. Scraping serves applications that need clean, typed fields.
  • Output: a crawler hands back URL lists and link graphs. A scraper hands back records you can load into a database.
  • Operational cost: crawling is limited by crawl budget and rate limits, since every discovered link is another request. Scraping is limited by cost per extraction, especially when rendering or LLM parsing is involved.

Most real pipelines need both. You crawl or map a domain to find the right pages, then scrape only those. Treating them as separate steps keeps costs legible, because you can see how much you spent finding pages versus extracting from them.

How crawling and scraping work technically

Under the hood, the two jobs share an HTTP layer but diverge fast. Google's Crawling and Indexing documentation describes the controls a site exposes, including robots.txt, sitemaps, and canonicalization. A developer typically implements these steps:

  1. Fetch and parse robots.txt before touching any path, following the parsing rules in RFC 9309. You can sanity-check a file with a free robots.txt validator.
  2. Seed the queue from the homepage and any sitemap, as covered in our sitemap discovery audit.
  3. Normalize and canonicalize links so tracking parameters and trailing slashes do not create duplicate URLs. Honor rel=canonical and meta robots signals.
  4. Choose a request strategy for each target: plain HTTP first, headless rendering only when the data is missing from the raw response.
  5. Extract to a schema, mapping page content into typed JSON, and return typed errors instead of empty results.
  6. Apply operational controls: per-host rate limits, backoff on rate-limit and server-error responses, and spending caps.

This is not legal advice, but a few guardrails matter for any developer building with web data. For deeper reading, see our breakdown of the legal axes that decide scraping legality.

  • Robots.txt is a request, not a license. RFC 9309 describes how crawlers are expected to behave, but it does not grant rights to reuse content. Courts have also found that failing to block crawlers does not create an implied license.
  • Fair use is fact-specific. The U.S. Copyright Office explains that fair use turns on four factors: purpose, nature, amount used, and market effect. Public availability alone does not permit unlicensed copying.
  • Commercial reuse carries risk. In Associated Press v. Meltwater, reproducing licensed news content for commercial gain did not succeed as fair use. Policy analysis also warns that RAG-style use of publisher content without a license can harm publishers' licensing markets.
  • Practical advice: get authorization for commercial uses, or lean on openly licensed sources.

Developer best practices checklist

A short list covers most of what separates a robust pipeline from a fragile one.

  1. Respect robots.txt and sitemaps, and crawl at polite rates. Back off on rate-limit and server-error responses, as outlined in our polite crawling defaults.
  2. Separate discovery from extraction so each step has its own cost and failure profile.
  3. Try HTTP-first parsing before reaching for a headless browser.
  4. Use schema-guided JSON extraction, validate every field, and stamp each record with its source URL and a timestamp.
  5. Set spending caps and return typed failures so an agent can reason about errors instead of guessing.

Pro Tip: Treat a blocked page as a distinct result, not an empty one. A typed "blocked" or "rate limited" response lets your pipeline retry correctly instead of silently storing nothing.

Tools and next steps for learning

The fastest way to learn the difference is to build it. Work through this sequence, and see our guide on scraping rate limits when you hit your first 429.

  • Build a tiny crawler: a seed URL, a queue, and robots.txt checks.
  • Scrape one page to JSON: pick a single page and pull three fields.
  • Validate the schema: reject records with missing or malformed values.
  • Explore library categories: HTTP clients, HTML parsers, headless browsers, and extraction frameworks, each with their own docs.

Crawl depth versus breadth strategies

A crawler has to decide how to spend a limited budget of requests, and the two classic strategies make opposite bets. Breadth-first crawling visits every link one hop from the seed, then every link two hops out, and so on. It surfaces shallow, high-level pages early, which is usually where a site's most important content lives, and it gives you a representative map quickly even if you stop partway.

Breadth-first and depth-first crawl comparison

Depth-first crawling follows one chain of links as far as it goes before backing up. It can reach deeply nested pages fast, but it tends to get trapped in endless pagination, calendar archives, or faceted filters, burning budget on near-duplicate URLs.

In practice most production crawlers are breadth-leaning with a hard depth cap. You set a maximum depth, a maximum page count per host, and rules that skip known traps such as session parameters. Sitemaps help here because they let you jump straight to deep pages that a pure link walk would reach late or never.

Pick depth when you need one specific section thoroughly, such as every page under a docs path. Pick breadth when you need coverage of an unknown site. Whichever you choose, a crawl cap is what keeps the bill predictable.

Performance and scalability: crawling versus scraping

The two jobs scale differently, and confusing them leads to bad capacity plans. Crawling is mostly I/O bound and politeness bound. The limiting factor is rarely your CPU. It is how fast a host will let you go before it starts returning 429 or 503 responses. So crawl throughput is governed by per-host concurrency, delays between requests, and how well your queue deduplicates URLs.

Scraping is bound by the cost of each extraction. Plain HTTP fetches plus selector parsing are cheap and parallelize well. Headless rendering is far heavier, because every page spins up a browser context. LLM-based extraction adds latency and per-call cost on top.

A few habits help both:

  • Cache aggressively. Do not refetch a page whose content has not changed.
  • Parallelize across hosts, not within one. Many sites at modest speed beats one site at full tilt.
  • Make failures cheap. Fail fast on dead links and surface a typed error rather than retrying blindly.
  • Meter by step. Tracking discovery spend separately from extraction spend tells you which half of the pipeline is growing.

At scale, predictable cost matters more than raw speed. A pipeline that is twice as fast but has an unbounded bill is harder to run than a slower one with a hard cap.

Handling dynamic content and JavaScript-heavy sites

Many modern pages ship a nearly empty HTML shell and fill it in with JavaScript. A plain HTTP request to such a page returns the shell, not the data, which breaks both crawling and scraping in different ways.

For crawlers, the problem is link discovery. If navigation is built client-side, the raw HTML may contain few or no links, so the crawl dies at the seed. Sitemaps are the escape hatch, since they list URLs regardless of how the page renders.

For scrapers, the problem is missing fields. The fix is usually layered:

  • Look for the underlying data first. Many pages load JSON from an API endpoint or embed it in a script tag. Reading that is faster and more stable than rendering.
  • Try HTTP-first, then escalate. Fetch the raw response, check whether your target fields exist, and only then fall back to headless rendering.
  • Wait for the right signal. When you do render, wait for a specific element or network condition rather than a fixed sleep.
  • Detect block and challenge pages. A bot-check page can look like a successful response. Classify it explicitly so it never gets stored as real data.

Rendering is a cost multiplier, so reserve it for pages that truly need it. A pipeline that renders everything by default pays for browsers it rarely needs.

Data quality and cleaning challenges

Crawling and scraping fail in different ways, so they leave different kinds of mess. Crawlers produce duplicates and noise: the same page under several URLs, tracking parameters, redirects, soft not-found pages that return a 200 status, and boilerplate such as navigation and cookie banners. Canonicalization and URL normalization are the first line of defense, along with honoring rel=canonical.

Scrapers produce schema problems. Fields go missing when a template changes, prices arrive with currency symbols or thousand separators, dates appear in inconsistent formats, and a selector that worked last month quietly starts grabbing the wrong element. Layout drift is the usual cause.

Practical cleaning habits:

  • Validate against a schema. Reject or flag records with missing, mistyped, or out-of-range fields instead of loading them.
  • Normalize early. Convert prices, dates, and units into one canonical form at ingestion.
  • Deduplicate by canonical URL and content hash, not by raw URL alone.
  • Stamp provenance. Store the source URL and fetch timestamp so you can judge freshness and trace errors.
  • Monitor for drift. A sudden spike in null fields usually means a page layout changed.

Clean data is cheaper than cleaned-later data. Catching a bad record at extraction time costs one retry. Catching it after it has fed a downstream model costs much more.

Why separate primitives matter for AI and RAG

When discovery and extraction are fused into one opaque call, you lose the ability to tell where money went or why quality dropped. We build around explicit primitives (Search, Traverse, Fetch, Extract, and Map) so an agent can reason about each step on its own terms. A failed fetch reads differently from a failed extraction, and the response says which one it was. That separation is also what makes predictable billing and typed failures possible.

— Glen

FAQ

What is the difference between web crawling and web scraping?

Crawling discovers and indexes pages by following links, producing URL lists and link graphs. Scraping extracts specific fields from pages into structured formats like JSON. Many pipelines use both: crawl to find pages, then scrape to pull the data.

Is AI scraping illegal?

Not automatically, but it depends on the facts. The U.S. Copyright Office describes fair use as a fact-specific four-factor test, and commercial reuse of licensed content carries real risk. Check the site's terms and seek authorization for commercial uses.

Can ChatGPT do web scraping?

A language model can help write scraping code or turn page text into structured fields, but it is not a crawler or fetcher on its own. You still need a reliable way to retrieve pages and handle rate limits, blocks, and errors. Schema-guided extraction pairs an LLM with that fetching layer.

Can I scrape data from any website?

No. Technical access does not equal permission. Review robots.txt, the site's terms, and the legal context, since lacking a robots.txt file is not an implied license. When in doubt, use openly licensed sources or ask for permission.

Sources

Run both primitives with predictable costs

If you want discovery and extraction as separate, typed calls with spending caps built in, we offer the Gyrence API with Search, Traverse, Fetch, Extract, and Map, plus a hosted MCP endpoint. Our Standard plan is $75 per month, Growth is $299 per month, and Scale is $549 per month, with Free, Founders, and Pay-As-You-Go options too. See every plan and start building on our pricing page.