MCP web scraping means exposing fetch, search, crawl and extract operations to an AI agent as tools over the Model Context Protocol, so the agent calls them directly instead of you wiring up scripts. It works well. It also introduces two problems scripts never had: scraped content can attack your model, and an agent in a retry loop can burn through a monthly budget in hours. This guide covers how to set it up so neither happens.
TL;DR
- An MCP scraping server gives an agent typed tools (search, fetch, extract, crawl, map) through one uniform protocol, hosted remotely or run locally.
- Treat every tool response as untrusted input. Prefer structured JSON or cleaned markdown over raw HTML, per OWASP guidance.
- Cap spend per mission, not just per account, and make failures explicit so agents stop instead of retrying blindly.
- Use an agent loop only where you need it. For static pages, a deterministic fetch is faster and cheaper.
Table of Contents
- What MCP changes about scraping
- The five operations an agent actually needs
- Security: the page is not your friend
- Cost: where agent scraping bills go wrong
- Failure modes your agent should be able to read
- Agent loop or plain fetch?
- Legal and ethical footing
- Our perspective
- Try it on a real workload
- FAQ
- Sources
What MCP changes about scraping
The Model Context Protocol is an open standard for letting a model call external tools in a uniform way. Claude's API can connect to remote MCP servers directly from the Messages API, without you building a separate client. That makes a hosted scraping endpoint a one-line addition to an agent rather than a custom integration.
The practical shift: the model, not your code, decides which URL to hit and when. That is the whole appeal, and the whole risk. A script fails the same way every run. An agent can decide to crawl 4,000 pages because a search result looked promising.
So the questions to ask of any MCP scraping setup are not about how many tools it has. They are:
- What does the agent see when a call fails?
- What does the agent see when a page is hostile?
- What stops one bad run from draining the budget?
The five operations an agent actually needs
Most agent web tasks decompose into the same five primitives. We built Gyrence around them, and they are exposed both as a plain API and through a hosted MCP endpoint:
| Primitive | What it does | Typical agent use |
|---|---|---|
| Search | Finds URLs for a query | Discovering sources |
| Traverse | Walks outward from a starting URL | Following a topic across a site |
| Fetch | Retrieves a page and cleans it to markdown | Reading one document |
| Extract | Returns JSON from a prompt or schema | Pulling fields, not prose |
| Map | Lists a domain's URL graph | Planning a crawl before spending on it |
The order matters. Map before you traverse. Fetch before you extract. An agent that can see the shape of a site first makes fewer expensive guesses.
Security: the page is not your friend
The most overlooked risk in MCP scraping is that a web page is attacker-controlled text that your model will read. OWASP describes MCP tool poisoning as an indirect prompt injection attack against agents connected to tool servers. Scraped HTML is the same vector from a different direction: a site can hide instructions in comments, hidden elements or alt text, and a model handed the raw markup may treat them as commands.
Three habits reduce the exposure:
- Return structured data where you can. If the agent needs a price and a date, an extraction call that returns typed JSON gives a hostile page far less room than a full HTML dump. See our guide to structured JSON extraction from the web.
- Validate in both directions. The OWASP MCP security cheat sheet says tool inputs should be treated as untrusted because they come from model output shaped by possibly malicious context. Outputs deserve the same suspicion.
- Tell the model what tool output is. State in the system prompt that tool return values are data, not instructions. It is not a complete defense, but it is cheap and it helps.
For a fuller walkthrough, our MCP security checklist turns the requirements into a week of work.
Cost: where agent scraping bills go wrong
Credit burn with agents is rarely linear. The classic failure is the runaway loop: the agent hits an error, retries, hits it again, and keeps going because nothing tells it to stop. A global monthly limit catches this only after the damage is done. What you want is a cap scoped to the task, so one confused run can spend its own allowance and no more.
That is the design behind our spending caps. A call reserves cost, settles on completion, and is refused once the cap is reached. The mechanics are in Reserve, Settle, Consume. Rate limits matter too: backing off for 20 seconds on a 429 and keeping a cap in place reduces both blocking risk and waste, as covered in our scraping rate limits post.
Two more cost rules we follow:
- No separate AI extraction charge. LLM extraction is bundled into the call's credit cost, so a schema-guided extract does not show up as a second line on the bill.
- Cost per call should be knowable before the call. If you cannot estimate a run's maximum cost from its parameters, you cannot safely hand it to an agent.
Failure modes your agent should be able to read
Most scraping tools return a page or an error string. An agent cannot reason about an error string. It guesses, and a guess is how you get a retry storm.
Every Gyrence call returns a typed, discriminated-union response, including the failures. The agent receives a distinct, machine-readable outcome and can branch on it:
- Success with content and source metadata
- Blocked or rate-limited, with a signal the agent can back off on
- Content that did not match the requested schema
- Cap reached, stop
The point is not that failures vanish. They do not. Some pages cannot be fetched, some sites block, some content is behind a login. The point is that the agent is told which, so it can stop, switch strategy or report back instead of fabricating an answer from nothing.
Agent loop or plain fetch?
Not every task needs an agent browsing. A 2026 benchmarking study on arXiv found that end-to-end agents help on sites with logins, while for static documentation or simple pages the agentic overhead adds latency and credit use compared with deterministic fetchers.
A workable rule:
| Situation | Use |
|---|---|
| Static docs, articles, product pages | Deterministic fetch, then extract |
| Whole-site ingestion for RAG | Map, then traverse with a cap |
| Login required | Browser automation for login, then plain HTTP, see headless browser scraping |
| Open-ended research across unknown sites | Agent with search plus fetch, hard cap per mission |
If you are unsure whether the pages you care about are reachable by automated clients at all, an AI crawlability audit shows what bots can and cannot access. And if the question is what an agent browsing actually involves, start with what agentic web browsing is.
Legal and ethical footing
Scraping law in the US is unsettled. The CFAA and trespass claims have been used by platforms to challenge scraping, and legal scholarship argues those causes of action should be limited in online copying cases. That is analysis, not a safe harbor, and the answer depends on facts: what is public, what the terms say, what you do with the data. Our write-up of four legal axes of scraping is a starting point; talk to a lawyer before building a product on contested data. Operationally, respect rate limits, identify your client honestly (see our user-agent reference), and keep PII out of what you store, as in ethical web scraping.
Our perspective
Web data is a storm. Pages change, block, lie and break, and most vendors hide that behind a clean success path. We think the opposite is right for agents: show the failure, because an agent that knows a fetch was blocked behaves better than one that was handed an empty string.
We also think pricing is part of reliability. A tool that is accurate but can bankrupt a side project in a weekend is not reliable. That is why caps are a primitive here, not a billing setting you find later, and why extraction costs no extra.
What we will not claim: that any scraper gets through every protected site. Nobody's does. We would rather you see the failure type and decide.
— Glen
Try it on a real workload
Gyrence offers a Free plan, Pay-As-You-Go metering, and monthly tiers: Standard at $75 per month, Growth at $299 per month and Scale at $549 per month. See the pricing page for what each includes, then connect the hosted MCP endpoint to your agent and run one capped mission. If you also need to know when a page changes, WebDoppler monitors pages and sends webhook alerts.
FAQ
Is MCP web scraping different from using a scraping API directly?
The underlying operations are the same. MCP changes who calls them: the model chooses tools and parameters at runtime. That makes typed errors, spending caps and output sanitization more important, because no human reviews each call.
Can a scraped page hijack my agent?
Yes, through indirect prompt injection. Hidden text on a page can read like an instruction to a model. Return structured JSON or cleaned markdown, treat tool output as untrusted, and follow the OWASP MCP guidance.
How do I stop an agent from running up a huge bill?
Set a cap per task, not only per account, and make sure failures return a clear stop signal. Without both, an agent that hits an error may retry until the monthly pool is empty.
Should I use an agent or a plain fetch for static pages?
Use a plain fetch. Benchmarks show agentic browsing adds latency and cost on simple pages, and its advantage shows up on sites that need logins.
Sources
- MCP Security Cheat Sheet, OWASP
- MCP Tool Poisoning, OWASP Foundation
- MCP connector, Anthropic docs
- Agent benchmarking study, arXiv
- Data Scraping as a Cause of Action, Fordham IPLJ