Claude MCP Scraping: Set Up Web Tools the Safe Way

Q: Why does my MCP server not show up in Claude Code?

The most common cause is a JSON entry with a url but no type, which is read as stdio and skipped. Add "type": "http". Then check claude mcp list and the init event's mcpservererrors field in recent versions.

To scrape with Claude, you give it a Model Context Protocol (MCP) server that does the fetching, and Claude decides what to do with the results. In Claude Code, that is one claude mcp add command. In Claude Desktop, it is a custom connector. The setup takes minutes. The hard part is what happens after: token costs, silent failures and a bill nobody capped.

This guide covers both clients, the config mistakes that quietly disable a server, and the controls to demand from any scraping MCP endpoint before you hand it to an agent.

Gyrence

Give Claude Structured Web Data

Gyrence connects to agents through a hosted MCP endpoint, with typed failure responses and spending caps for web data workflows.

Explore Gyrence

TL;DR

Claude is the brain and the MCP server is the hands: the server fetches, renders and cleans pages, and Claude reasons over the output.
In Claude Code, add a remote server with claude mcp add --transport http <name> <url>. A JSON entry with a URL but no type gets skipped.
Claude Desktop connectors run from Anthropic's cloud, so the server must be reachable from the public internet.
Return clean markdown or schema-shaped JSON, not raw HTML, and cap spend per workspace.
Pick a server whose failures are typed, so Claude can tell a blocked page from an empty one.

How Claude uses a scraping server
Claude Code: add the server
Claude Desktop: custom connector
Where the money goes
Failure modes your server should name
Security before you install
Our view: what we'd check first
Try it on your own workspace
A short test plan before production
FAQ
Sources

How Claude uses a scraping server

A practical mental model from 2026: Claude decides what to extract and structures it, while a scraping API or MCP server handles fetching, JavaScript rendering and anti-bot work. Claude does not browse by itself.

MCP is the contract between them. Servers expose tools that a model can invoke, and the protocol runs on JSON-RPC 2.0. For scraping, a good tool set maps to distinct jobs:

Job	Primitive	What Claude gets back
Find pages	Search	Ranked URLs with titles and snippets
Read one page	Fetch	Cleaned markdown
Pull fields	Extract	JSON matching your schema
Walk a site	Traverse	Pages discovered outward from a start URL
Plan a crawl	Map	The domain's URL graph

Separate tools let the model pick the cheapest one that answers the question. A model that can only "scrape" will fetch whole pages when it needed one field.

Claude Code: add the server

The Claude Code docs recommend remote HTTP servers for cloud services. The syntax:

claude mcp add --transport http <name> <url> \
  --header "Authorization: Bearer <your-api-key>"

The endpoint URL and key come from your provider's console. For Gyrence, the connect page shows how to attach an agent to our hosted MCP endpoint.

Scope matters. A server can live in ~/.claude.json for every project, or in a project-level .mcp.json you commit for your team. Keep the API key out of the committed file.

The config error that skips your server

If you write JSON by hand, set type explicitly. The docs state that an entry with a url but no type is read as a stdio server, which is a configuration error, and Claude Code skips it. streamable-http works as an alias for http. Newer Claude Code versions report skipped servers through the mcp_server_errors field of the system/init event, so check there before blaming the scraper.

A working .mcp.json entry looks like this:

{
  "mcpServers": {
    "web-data": {
      "type": "http",
      "url": "<endpoint-from-your-console>",
      "headers": { "Authorization": "Bearer ${WEB_DATA_API_KEY}" }
    }
  }
}

Then run claude mcp list and ask Claude to call one tool against a page you know. If the tool does not appear, the server was skipped or auth failed.

Claude Desktop: custom connector

Desktop works differently, and this trips people up. When you add a custom connector, Claude connects to your remote MCP server from Anthropic's cloud, not from your machine. Servers behind a VPN, on a private network or blocked by a firewall will not connect, even if you can reach them locally. Free users are limited to one custom connector.

So for Desktop:

Open the connector settings and choose to add a custom connector.
Paste the remote MCP URL from your provider.
Complete the sign-in or key step the provider requires.
Start a chat, enable the connector, and ask for a single page as markdown.

If you need a localhost scraper, run it through Claude Code's stdio transport instead.

Where the money goes

Scraping through an agent has two cost layers: the scraper's credits and the LLM's input tokens. According to webscraping.cc, token costs for web content often dominate the total, and clean markdown without scripts and styles costs far less than raw HTML. The same source says prompt caching can cut the cost of repeated schema instructions by roughly 90% for those segments.

Practical consequences:

Fetch as markdown by default. Our HTML vs. markdown breakdown shows where the savings come from.
Extract when you need fields. Ask for JSON against a schema and Claude reads a few hundred tokens instead of a whole page. See our structured JSON extraction guide.
Map before you traverse. Look at the URL graph first, then crawl only the sections you want.
Cap the workspace. An agent in a loop is the classic way to a surprise invoice.

Failure modes your server should name

Most scraping tools return a string. When the page is blocked, empty, redirected to a login wall or rendered client-side, Claude gets text that looks like content and reasons over it anyway. That is how an agent confidently summarizes a cookie banner.

Demand typed responses. A discriminated union, where every call returns a status that is ok, blocked, not_found, timeout or similar, lets the model branch: retry, try another source, or tell you it could not read the page. Silence is the failure mode to avoid. In compliance and financial research, a missing source should be reported as missing.

Security before you install

Treat every MCP server as code with reach. One tested roundup advises starting with read-only servers, using per-project API keys, limiting access to dev or test data, and logging what gets called. Scraping servers are read-mostly, which helps, but fetched pages can contain text that tries to instruct your agent. Keep write-capable tools (shell, database, email) behind approval prompts when scraped content is in context.

Five safeguards for MCP server security

If your agent's job is SEO rather than general scraping, a purpose-built Claude MCP for SEO data is a separate tool from a web-data server, and the two can sit side by side.

Our view: what we'd check first

Web data is a storm: pages change, block you, redirect and lie by omission. We built Gyrence around the assumption that the agent needs to know which of those happened.

Our view: what we'd check first — overview diagram

In practice that means five composable primitives (Search, Traverse, Fetch, Extract, Map) behind one API and one hosted MCP endpoint, with typed responses for failures as well as successes. LLM extraction is bundled, with no separate AI extraction charge, and spending caps sit at the workspace level. Our rule when evaluating any scraping MCP, ours included: run it against a page that blocks you and a page that does not exist, and read what comes back. If you cannot tell the two apart, the agent can't either.

For wiring details, our guides on setting up an MCP server and connecting a scraping API to an agent go step by step. Building on LlamaIndex instead? See LlamaIndex MCP.

— Glen

Try it on your own workspace

Gyrence has a Free plan, Founders and Pay-As-You-Go options, and paid tiers: Standard at $75 per month, Growth at $299 per month and Scale at $549 per month. Compare them on the pricing page, then point Claude at the hosted endpoint. If you need change alerts on a page rather than one-off reads, WebDoppler monitors with webhooks.

A short test plan before production

Fetch a normal article as markdown and check that navigation and scripts are gone.
Fetch a URL that returns 404 and confirm the response says so explicitly.
Fetch a JavaScript-heavy page and compare against what you see in a browser.
Run an extraction with a schema and verify null fields come back null, not invented.
Set a low spending cap, loop the agent deliberately, and confirm it stops.

The MCP spec is also moving toward long-running tasks, so favor servers that can handle asynchronous crawls rather than only one-shot fetches.

FAQ

Does Claude Desktop need a local server for scraping?

No. Custom connectors use a remote MCP server that Claude reaches from Anthropic's cloud. A local-only server will not work there unless it is publicly reachable. For local tools, use Claude Code with stdio.

Why does my MCP server not show up in Claude Code?

The most common cause is a JSON entry with a url but no type, which is read as stdio and skipped. Add "type": "http". Then check claude mcp list and the init event's mcp_server_errors field in recent versions.

Should Claude read raw HTML or markdown?

Markdown for reading, structured JSON for specific fields. Raw HTML burns input tokens on scripts and styles that carry no meaning, and token cost is often the largest part of an agent scraping bill.

How do I stop an agent from running up a scraping bill?

Use a provider with workspace spending caps and predictable per-call credit costs, and test the cap deliberately. Also prefer Map and Extract over broad crawls, so the agent fetches only what it needs.