An LLM-friendly scraper does three things. It returns clean, structured content instead of raw HTML. It tells your agent exactly what happened on every request, including the failures. It keeps cost and provenance predictable enough that you can run it unattended. Most tools nail the first and skip the other two.
This guide gives you six criteria to test any scraper against, a short legal checklist, and the way we approach each at Gyrence.
TL;DR
- Raw HTML wastes tokens and hurts citation accuracy. Look for clean markdown for prose and typed JSON for fields.
- A scraper an agent can trust returns typed failures (blocked, empty, timed out), not a 200 with nothing useful in it.
- Cost has to be capped and predictable. An agent in a retry loop can burn a budget overnight.
- Keep the source URL and fetch time with every record. Provenance is what makes the data defensible later.
- "Public" does not mean "free to use for anything." Check terms, robots.txt and your own privacy commitments.
Table of Contents
- Why raw HTML is a poor input for a model
- Six criteria for an LLM-friendly scraper
- A quick comparison by type of tool
- The legal checklist before you ingest
- Our perspective: honesty is the feature
- Trying it on your own pages
- FAQ
- Sources
Why raw HTML is a poor input for a model
A typical page is mostly navigation, scripts, cookie banners and ad slots. The article you wanted is a fraction of the bytes. Feed all of it to a model and you pay for tokens that carry no signal, and you push the useful text further from the question in the context window.
The research points the same way. TREC 2025 work on retrieval-augmented generation, as collected in NIST's browser for the DRAGUN track, reflects a shift from simple fetch-and-extract toward agentic RAG. Models struggle with structural complexity such as dynamic JavaScript, so the scraping layer has to hand over semantically rich, cleaned data if you want answers that cite accurately.
That is the first requirement: the scraper does the cleaning, not the prompt. For a deeper look at what that cleaning involves, see our notes on what clean page markdown means and why markdown output matters for scraping.
Six criteria for an LLM-friendly scraper
Use these as a checklist when you evaluate a hosted API, an open-source library, or a script you wrote yourself.
1. Format-aware output
Prose belongs in markdown: headings, lists and tables preserved, boilerplate stripped. Fields belong in JSON: price, author, date, SKU. A scraper that offers only one of these pushes the other job onto your model, where it is slower, costlier and less repeatable.
Check what happens to tables and code blocks. Flattened tables are a common silent failure, and they change meaning. We cover the tradeoffs in preserving HTML during content normalization for RAG.
2. Schema-guided extraction
When you need fields, you want to hand over a prompt or a JSON schema and get back an object that validates against it. Two things matter here:
- The result is typed, so your code can branch on it.
- A field the page does not contain comes back as missing, not as a plausible invention.
Ask any vendor what happens when the page lacks the requested field. The answer tells you whether extraction is honest.
3. Typed, honest failures
This is the criterion most tools skip. A human sees a CAPTCHA page and moves on. An agent sees text and may summarize the CAPTCHA.
An LLM-friendly scraper returns a discriminated union: a success variant with content, and distinct failure variants for blocked, not found, timed out, empty or wrong content type. Your agent can then reason about the result (retry later, try another source, tell the user) instead of guessing.
Test it directly. Point the scraper at a login wall, a 404, a PDF and a JavaScript-only page. Note what you get back for each.
4. Cost that cannot run away
Agents loop. A planner that fails and retries 400 times is a normal bug, and on usage-based billing it becomes a bill. Look for hard spending caps, a fixed cost per call, and no separate surcharge for the LLM step inside extraction. If the pricing page needs a spreadsheet to interpret, the invoice will too.
5. A protocol your agent already speaks
If your agent runs in a framework or an MCP client, the scraper should show up as tools the model can call directly, with typed inputs and outputs. A hosted Model Context Protocol endpoint removes the glue code and the schema drift that comes with hand-wrapped HTTP calls.
6. Provenance on every record
Every result should carry the source URL, the fetch time and enough metadata to reproduce the claim later. If a compliance reviewer asks where a number came from, "the agent found it" is not an answer.
A quick comparison by type of tool
| Criterion | Script with an HTML parser | Browser automation | Open-source LLM crawler | Hosted agent-oriented API |
|---|---|---|---|---|
| Clean markdown | You build it | You build it | Built in | Built in |
| Typed failures | Whatever you write | Whatever you write | Varies | Should be built in |
| JavaScript pages | No | Yes | Yes | Yes |
| Spending caps | N/A, but you run the infrastructure | N/A, same | N/A, same | Check the plan |
| Ops burden | Low at first, grows | High | Medium to high | Low |
Self-hosting makes sense when you have few, stable sources and the engineering time. If you go that route, our guide to scraping dynamic pages with Python and Playwright walks through the rendering side. Past a handful of sources, maintenance tends to cost more than the API.
The legal checklist before you ingest
We are engineers, not your lawyers. For anything beyond internal experiments, have counsel review your use case. These are the points that come up most.
Terms of use and bypassing barriers. Legal commentary from Ropes & Gray argues the line is not whether a site is public, but how the data is used and what you bypassed. Their guidance suggests treating your pipeline as a contractual supply chain, vetting sources for copyright and licensing risk before ingestion.
robots.txt. The Robots Exclusion Protocol is standardized as RFC 9309, and it is how site owners tell crawlers what they may access. Honor it by default. Publishers who want to restrict AI crawlers while staying discoverable have real options, and it helps to understand how they block AI crawlers selectively so you can read their signals correctly.
Training is not the same as storing. In 2025, two Northern District of California rulings treated training on books as transformative. But the court in Bartz v. Anthropic separated that from downloading and keeping pirated copies in a central library, and Kadrey v. Meta turned on its own facts. Source provenance matters. These are district court decisions, not settled law across the US.
Your own promises. The FTC has warned that AI companies must uphold their stated privacy and confidentiality commitments when they use data to build or refine models. If your policy says one thing and your pipeline does another, "it was publicly accessible" will not help you.
For the engineering side, we have written up four legal axes that decide scraping legality, ethical scraping with rate limits, schemas and PII controls, and five audit-ready controls.
Our perspective: honesty is the feature
We named the product's design principle before we wrote much code: honest by design. Web data is a storm. Pages change, block, time out and lie. A scraper that hides this to look reliable moves the damage downstream, into your model's answer.
So every Gyrence call returns a typed response, failures included. The five primitives are Search, Traverse, Fetch, Extract and Map, and each can be composed with the others. Fetch cleans a page to markdown. Extract takes a prompt or schema and returns structured JSON. Map lays out a domain's URL graph. Traverse walks outward from a starting URL. An agent can tell "the page is empty" from "we were blocked" from "the field does not exist."
We also think pricing is part of reliability. Extraction uses an LLM, and we bundle it: there is no separate AI extraction charge, and spending caps stop a runaway loop before it becomes an invoice. Our claim is narrow. We do not promise to get past every defense, and we do not promise every page returns data. We promise that when it does not, you will know why.
— Glen
Trying it on your own pages
A fair test takes an afternoon. Pick ten URLs that worry you: a JavaScript-heavy page, a table-heavy page, a paywalled one, a dead link. Run them through each candidate and compare the output and the failure responses.
Gyrence has a Free plan, a Pay-As-You-Go option, and monthly tiers: Standard at $75 per month, Growth at $299 per month and Scale at $549 per month. The pricing page lists what each includes. If your use case is watching pages for changes rather than one-off fetches, WebDoppler handles monitoring with webhook alerts. To see the markdown side first, read our guide to fetching web pages to markdown for an LLM.
FAQ
Is a markdown converter the same as an LLM-friendly scraper?
No. A converter handles one step: turning HTML into readable text. An LLM-friendly scraper also handles fetching, rendering, extraction to a schema, typed errors and cost control. If your sources are static and few, a converter may be enough.
Can I just ask an LLM to scrape a page?
You can, but it is the wrong tool for the fetch. A model reading raw HTML is slow, costly and can invent fields that are not there. Let a scraper retrieve and clean the page, then use the model for the part it does well: interpreting the content or filling a schema.
Is AI scraping illegal?
Not as a blanket rule. Legality depends on the site's terms, what technical barriers you bypass, the kind of data (especially personal data) and how you use it. Courts have so far looked at training and storage separately. For a commercial pipeline, get legal advice on your specific sources.
How do I stop an agent from overspending on scraping?
Set a hard spending cap at the account or workspace level, prefer fixed cost per call, and cap retries in your agent code. Do not rely on a dashboard you check weekly.
Sources
- NIST TREC 2025 DRAGUN proceedings
- Ropes & Gray: Web Scraping in the Age of AI
- RFC 9309: Robots Exclusion Protocol
- Bartz v. Anthropic summary, U.S. Copyright Office
- Kadrey v. Meta summary, U.S. Copyright Office
- FTC: AI Companies, Uphold Your Privacy and Confidentiality Commitments