An agent search API returns structured results and page content that an AI agent calls programmatically, typically through search and fetch primitives exposed over a direct API or a Model Context Protocol (MCP) server. Reach for live search when an agent needs current facts, prices, or citations; for testing and benchmarks, frozen or cached environments like RetroSearch keep results reproducible.
TL;DR:
- Reserve live search for facts, prices, and availability that may change; use cached results for development and repeated runs, and frozen snapshots for benchmarks.
- Load a lightweight tool catalog first, inspect a schema only after choosing a capability, then execute calls through code that returns compact results.
- Freeze pages or record real calls for deterministic CI replay, then score outputs and gate merges; also verify canonical URLs and track citation accuracy.
- Start with a hosted MCP server for a quick integration; move to a direct API when caching, failure handling, or cost requires tighter control.
Table of Contents
- What is an agent search API (primitives and response shapes)
- Live search versus cached results: when reproducibility wins
- How MCP servers expose search and fetch tools
- Progressive discovery and programmatic tool calling
- Testing web-enabled agents with RetroSearch and deterministic replay
- Quickstart: request patterns, modes, and token budgets
- Choosing an agent search API: a practical checklist
- Pragmatic advice for production agent builders
- Give your agents reliable web access with Gyrence
- FAQ
- Sources
What is an agent search API (primitives and response shapes)
An agent search API exposes a small set of callable primitives rather than a single endpoint. Two common primitives are:
- Search: takes a query and returns a results array with
url,title,snippet, and metadata like a crawl timestamp or content type. - Fetch: takes a URL and returns the full document, often normalized to Markdown for easier parsing by a language model.
Typed, discriminated-union responses matter because an agent has to branch on outcomes without guessing. A result that includes a canonical URL, a content type, and a clear success or failure flag lets an agent decide whether to cite a source, retry, or fall back to another tool. The MCP specification standardizes this discovery and invocation layer so agents can implement one search and fetch interface that works across different servers and environments, instead of hand-rolling a parser per vendor.
Live search versus cached results: when reproducibility wins
Live search earns its cost when freshness matters: breaking news, price and availability checks, or validating a claim an agent is about to make. Cached or frozen search earns its cost everywhere else, especially testing.
- Use live mode for anything time-sensitive where a stale answer is a wrong answer.
- Use cached mode for development, demos, and anything you plan to run more than once.
- Use a frozen snapshot for benchmarks where the web itself must not move between runs.
Deep Research Bench built RetroSearch, a frozen web snapshot environment, specifically because the live web changes daily and undermines repeatable evaluation. Every live call also adds tokens to the model's context window, so teams weighing latency and cost against freshness often default to cached results until a query genuinely needs the current web.
How MCP servers expose search and fetch tools
A hosted MCP server advertises its tools through a discovery flow before an agent ever calls one. A client checks /.well-known/mcp and server cards, then calls tools/list to see what is available before invoking anything, which keeps the initial context small.
- Discovery:
/.well-known/mcpand server cards tell a client what a server offers before any tool call happens. - Tool contracts: a
searchtool returns a results array; afetchtool returns full document content plus a canonical URL so the agent can cite its source accurately. - Access modes: a public
/mcpendpoint handles unauthenticated or low-trust access, while/authed/mcpsupports client-credential flows for programmatic, production-grade access.
Returning a canonical URL on fetch is not a nicety. An agent that cites a redirect target or a tracking link instead of the real source produces citations nobody can verify, which defeats the point of grounding a response in web data in the first place.
Progressive discovery and programmatic tool calling
Loading every available tool definition into context up front wastes tokens and slows the model down before it has decided what it needs. MCP's own best-practices guidance recommends a layered approach instead:
- Catalog: expose a lightweight
search_toolscall that lists available tools by name and short description, nothing more. - Inspect: fetch the full schema for a tool only once the agent has decided it needs that specific capability.
- Execute: call the tool, ideally through sandboxed generated code that can chain multiple calls and return a small, final result to the model rather than dumping intermediate output into context.
This three-layer pattern, confirmed in MCP's client best practices, keeps token and latency costs down because the model never sees a schema it doesn't need. Group tools by server, cache tool definitions across requests, and use keyword, embedding, or hybrid search over your tool catalog once it grows past a handful of entries.
Pro Tip: Cache tool schemas client-side and invalidate them on a version bump instead of refetching tools/list on every session.
Testing web-enabled agents with RetroSearch and deterministic replay
Web-enabled agents fail quietly. A search provider changes its ranking, a page layout shifts, or a snippet format drifts, and an agent's grounding degrades without throwing an error anyone notices.
- RetroSearch: freeze a set of web pages into a stable snapshot so a benchmark run produces the same inputs every time, regardless of what the live web looks like that day.
- Capture, replay, score, gate: record real tool calls and responses, replay them deterministically in CI, score outputs with an LLM-judge ensemble, and gate merges on a score threshold.
- CI checklist: pin search mode to cached or frozen for test suites, assert on canonical URLs in fetched results, and track citation accuracy over time rather than just pass or fail.
A documented capture, replay, score, gate pipeline uses deterministic replay harnesses alongside LLM-judge ensembles specifically to catch regressions before they reach production. Treat this the same way you'd treat a snapshot test for any other interface contract.
Quickstart: request patterns, modes, and token budgets
A minimal agent tool entry for web search typically looks like a web_search block inside the agent's tools array, with a mode field controlling behavior:
{
"type": "web_search",
"mode": "live",
"context_size": "medium",
"allowed_domains": ["example.com"]
}
modeaccepts live, cached, or disabled, and should default to cached outside of production traffic.context_sizecontrols how much snippet and metadata text gets injected per result, directly trading recall for token cost.allowed_domainsrestricts search to trusted sources, which cuts noise and reduces the chance of an agent citing an unreliable page.
This pattern is consistent with how OpenAI's web search tool structures its mode, context_size, and allowed_domains settings. Keep result limits small, stream fetch output where the client supports it, and set a hard ceiling on total fetched tokens per turn so one noisy query can't blow your context budget.
Pro Tip: Start every new integration with mode: "cached" and allowed_domains set, then loosen both once you've confirmed the agent's behavior is predictable.
Choosing an agent search API: a practical checklist
Weigh any provider against the same operational questions, not just raw search quality:
- Does it support MCP directly, or only a proprietary SDK?
- Does it expose both search and fetch as separate, composable primitives?
- Are failure responses typed rather than generic error strings?
- Does it offer structured extraction, not just raw HTML or Markdown?
- Is pricing predictable, with caps rather than open-ended metered surprises?
- Does it support monitoring or webhooks for detecting content drift?
| Evaluation area | What to check in docs |
|---|---|
| Protocol support | MCP discovery endpoints and tool contracts |
| Response typing | Discriminated-union success and failure shapes |
| Reproducibility | Cached or frozen search modes for testing |
| Pricing | Credit caps versus uncapped metered billing |
| Monitoring | Webhook-based drift or change alerts |
Reproducibility deserves more weight than most teams give it. A provider that can't return consistent results for a frozen test suite will quietly break your CI the moment its underlying search index changes.
Pragmatic advice for production agent builders
Start with a hosted MCP server when you want search and fetch working in an afternoon; integrate a search API directly only once you need tighter control over caching, failure handling, or cost. Begin with a minimal progressive discovery surface, two or three tools, and add more only as real agent behavior demands it.
For teams that want search, fetch, and structured extraction under one roof instead of stitching together separate vendors, Gyrence's primitives (Search, Traverse, Fetch, Extract, Map) cover that ground without extra integration work.
— Glen
Give your agents reliable web access with Gyrence
Building agent search from scratch means stitching together a search provider, a fetch and normalization layer, an extraction pipeline, and monitoring for when source pages drift. We built Gyrence with composable primitives for Search, Traverse, Fetch, Extract, and Map, plus a hosted MCP endpoint, so your agent gets all of that through one typed interface instead of separate vendor contracts.
Every call returns a discriminated-union response, including the failure cases, so your agent code can branch on real outcomes instead of parsing error strings. Extraction is schema-guided and LLM-powered with no separate line item on your invoice, and spending caps keep usage-based billing from turning into a surprise at the end of the month. WebDoppler adds webhook alerts when a monitored page changes, which pairs naturally with the drift-detection checklist above.
- Search and Fetch: the same core primitives this guide describes, available over MCP or direct API.
- Extract: schema-guided JSON extraction bundled into the same call, no add-on pricing.
- Traverse and Map: crawl outward from a URL or map a domain's structure for site-wide ingestion.
- WebDoppler: webhook-based monitoring for content drift and change detection.
Read the docs to see request shapes for each primitive, or check pricing to compare the Free, Founders, Pay-As-You-Go, Standard, Growth, and Scale plans before connecting your first agent.
FAQ
What is an agent search API used for?
An agent search API lets an AI agent query the live web and fetch page content programmatically, typically returning structured results with URLs, titles, and metadata. Agents use it for current facts, citations, and lookups that a static training dataset can't cover.
Should I use live or cached search mode by default?
Use cached mode for development and testing, since it keeps results stable across repeated runs, and reserve live mode for production queries that need current information. The OpenAI web search tool exposes both as explicit mode options for exactly this reason.
What is RetroSearch and why does it matter for testing?
RetroSearch is a frozen web snapshot environment introduced by Deep Research Bench to make agent evaluation repeatable despite the live web changing constantly. It lets teams rerun the same benchmark query and get the same underlying page content each time.
How does MCP reduce token usage for agent tools?
MCP's client best practices recommend progressive discovery: an agent loads a minimal tool catalog first and only fetches a full tool schema once it has decided to use that tool. This keeps unnecessary schema data out of the model's context window.
Does Gyrence support MCP for agent integrations?
Yes, Gyrence hosts an MCP endpoint alongside its direct API, so an agent can call Search, Traverse, Fetch, Extract, and Map primitives through either interface. Details on both connection methods are in the docs.