Unstructured HTML is defined as raw, layout-heavy markup that mixes navigation, scripts, tracking code, and content into a single undifferentiated payload. This is the primary reason why agents fail on unstructured HTML: they receive noisy, semantically ambiguous input that wastes token budget and produces interaction errors. LLM agents, DOM-based scrapers, and accessibility tree parsers all degrade when fed pages where 80–90% of the markup is boilerplate rather than meaningful content. The industry term for this problem is "unstructured web data," and solving it requires rethinking what you feed your agent before it ever takes an action.
Why agents fail on unstructured HTML: the core failure modes
Agents fail on unstructured HTML for three distinct technical reasons: hallucinated interactions, DOM fragility, and state blindness. Each one compounds the others.
Hallucinated interactions occur when noisy DOM structure gives the agent ambiguous signals about what is clickable, readable, or actionable. Research shows that 34% of agent actions cause no DOM changes at all, meaning the agent clicked or submitted something that had no effect. The agent has no way to detect this without explicit feedback, so it either retries indefinitely or proceeds on a false assumption.
DOM fragility is the second failure mode. Agents that rely on CSS selectors or XPath expressions break silently when a site updates its layout. A routine front-end deploy can rename a class or restructure a div, and the selector that worked yesterday now returns nothing. Screenshot-based vision models add another layer of fragility: even a 10-pixel UI layout shift causes incorrect click targeting, with no error raised.
State blindness is the most underappreciated failure mode. A DOM snapshot captures the HTML at a single moment. It has no awareness of scroll position, authentication state, or whether client-side JavaScript has finished hydrating the page. Agents acting on skeleton placeholders or pre-hydration stubs will hallucinate element existence and attempt interactions that are structurally impossible.
- 10.4% of agent failures come from repetitive failed actions
- 34% of actions cause zero DOM changes
- 16% of actions target elements outside the intended domain
- 32% of navigation attempts follow nonexistent links
These numbers are not edge cases. They describe the baseline behavior of agents operating on unstructured pages.
How does raw HTML token overhead affect agent reasoning?
Raw HTML inflates token usage far beyond what the actual content requires. Preprocessing HTML into clean Markdown can reduce token usage by up to 70%. A 12,000-token raw page can shrink to 3,500 tokens after cleaning. That reduction directly cuts inference costs and improves the quality of reasoning because the model spends its context window on signal, not noise.
The problem is structural. Navigation menus, cookie banners, footer links, inline scripts, and analytics tags all appear in the raw HTML payload. The LLM has no reliable way to distinguish these from primary content. Models frequently misinterpret navigation and footer content as the main site text, which corrupts retrieval results in RAG pipelines and produces hallucinated summaries.
| Format | Approx. token count | Reasoning quality | Cost impact |
|---|---|---|---|
| Raw HTML | ~12,000 | Low, noisy context | High |
| Cleaned Markdown | ~3,500 | High, focused context | Low |
| Typed JSON | ~2,000–4,000 | Highest, typed fields | Lowest |
Pro Tip: Before feeding any page to an LLM agent, strip everything outside the <main> tag or its semantic equivalent. Navigation, scripts, and footers are almost never relevant to the agent's task.
Markdown compression does introduce its own tradeoffs. Round-trip parsing from Markdown back to structured code produces 80% ambiguity, which matters when agents need to generate or validate interactive output. Clean Markdown is the right format for reasoning. It is not always the right format for output.
What extraction techniques actually improve agent performance?
The industry is moving away from raw DOM and screenshots toward tokenized accessibility trees that reduce failure modes and improve semantic clarity. An accessibility tree exposes element roles, labels, and interactive states without layout noise. The browser enforces these roles natively, so the agent gets a functionally strict interaction model rather than an ambiguous pile of div elements.
Structured JSON extraction takes this further. Typed fields in JSON output eliminate the expensive post-processing regex work that raw HTML requires. When the agent receives {"price": 29.99, "in_stock": true} instead of a <span class="price">$29.99</span> buried in a table, downstream logic becomes deterministic. You can validate against a schema. You can catch failures at the type level.
| Approach | Pros | Cons |
|---|---|---|
| Raw HTML | No preprocessing needed | Noisy, token-heavy, fragile |
| Cleaned Markdown | Token-efficient, readable | Round-trip ambiguity for interactive output |
| Accessibility tree | Semantic roles, low noise | Requires browser runtime |
| Typed JSON | Deterministic, schema-validatable | Requires extraction prompt or schema |
Pro Tip: Use a structured JSON extraction approach for any agent task that requires downstream logic. Reserve Markdown for tasks where the agent only needs to read and reason.
Hybrid pipelines combine these formats. The agent uses Markdown or an accessibility tree for reasoning and navigation, then switches to JSON for data capture. This pattern balances token efficiency with output reliability.
How can developers reduce agent failures on dynamic web pages?
The most effective strategy is aggressive content extraction before the agent ever sees the page. Strip boilerplate, extract the main content region, and convert to a clean format. This single step eliminates the majority of token waste and removes the navigation noise that confuses LLM reasoning.
Selector stability is the next priority. CSS selectors and XPath break silently after routine site updates. Stable identifiers like data-* attributes and accessibility roles survive front-end refactors far better than class names or positional selectors. Build your extraction logic around these stable anchors. You can also use domain URL maps to give agents a reliable navigation graph rather than letting them discover links from noisy page content.
Asynchronous page states require explicit handling. Do not snapshot a page immediately after load. Wait for hydration signals, network idle events, or specific element presence before capturing the DOM. Agents acting on pre-hydration HTML will encounter elements that do not exist in the rendered page.
- Extract main content only. Remove nav, footer, scripts, and tracking before passing to the agent.
- Use
data-*attributes and ARIA roles as selectors. Avoid class names and positional XPath. - Wait for full hydration before snapshotting. Use network idle or element-present signals.
- Validate structured outputs against a schema. Catch type errors before they reach downstream logic.
- Log every no-op action. If an agent action causes zero DOM changes, treat it as a failure signal.
Pro Tip: For RAG web ingestion pipelines, always store both the cleaned Markdown and the source URL. Retrieval quality degrades when chunks lose their provenance.
Key Takeaways
Agents fail on unstructured HTML because raw markup is token-heavy, semantically ambiguous, and state-blind, and fixing this requires structured extraction before the agent ever acts.
| Point | Details |
|---|---|
| Raw HTML is mostly noise | Up to 90% of a raw page is boilerplate that inflates token costs and corrupts agent reasoning. |
| State blindness causes hallucinations | DOM snapshots miss hydration, scroll, and auth state, leading agents to act on elements that do not exist. |
| Markdown cuts token usage sharply | Cleaning HTML to Markdown can reduce token count by up to 70%, lowering cost and improving retrieval quality. |
| Typed JSON is the most reliable output | Structured JSON with typed fields removes post-processing regex work and enables schema validation. |
| Stable selectors prevent silent failures | Using data-* attributes and ARIA roles instead of CSS class names survives routine site updates. |
The real cost of ignoring HTML structure
Most developers I've worked with treat HTML cleaning as a preprocessing detail. They assume the LLM will figure it out. It won't, not reliably, and the failure modes are silent. The agent doesn't throw an exception when it clicks a skeleton placeholder. It just moves on with a broken assumption baked into its state.
The shift to accessibility trees is not a minor optimization. It is a fundamentally different contract between the agent and the page. Raw DOM gives the agent a visual approximation. An accessibility tree gives it a functional specification. That difference shows up in production as the gap between an agent that works 60% of the time and one that works 95% of the time.
The developers who get this right early build extraction pipelines that are format-aware from the start. They don't bolt on cleaning as an afterthought. They treat the output format as part of the agent's interface contract, the same way you'd type a function signature before writing the body.
— Glen
Gyrence handles the extraction layer so your agents don't have to
Raw HTML is a solved problem at the infrastructure level. Agents should never receive unstructured page content directly.
Gyrence turns any web page into clean, agent-ready structured data through five composable API primitives: Search, Traverse, Fetch, Extract, and Map. The Fetch primitive returns cleaned Markdown. The Extract primitive returns typed JSON against your schema or prompt. Every response is a typed, discriminated-union, including the failure cases, so your agent reasons about results instead of guessing. Spending caps mean your scraping bill stays predictable. The hosted MCP endpoint connects directly to agent frameworks without custom glue code. Start at gyrence.com.
FAQ
Why do agents fail on unstructured HTML?
Agents fail because raw HTML mixes boilerplate, scripts, and navigation with actual content, wasting token budget and producing ambiguous input. Up to 90% of a raw page is layout scaffolding that has no value for agent reasoning.
What is DOM state blindness in agent interactions?
DOM state blindness means a page snapshot has no awareness of scroll position, authentication state, or JavaScript hydration. Agents acting on pre-hydration snapshots interact with elements that do not exist in the rendered page.
How much does cleaning HTML reduce token usage?
Preprocessing raw HTML into clean Markdown can reduce token usage by up to 70%. A 12,000-token raw page can shrink to approximately 3,500 tokens after boilerplate removal.
What is an accessibility tree and why does it help agents?
An accessibility tree exposes element roles, labels, and interactive states without layout noise. It gives agents a functionally strict interaction model that reduces hallucinated clicks and failed actions.
Should agents use Markdown or JSON for web data?
Use Markdown when the agent needs to read and reason. Use typed JSON when the agent needs to capture data for downstream logic. Hybrid pipelines that combine both formats deliver the best balance of token efficiency and output reliability.