Visit Website →
← Back to blog

Outlink Crawling: Fix Failures in Under 10 Minutes for Developers

September 26, 2026
Outlink Crawling: Fix Failures in Under 10 Minutes for Developers

An outlink can be extracted during parsing and still never get fetched. Extraction, policy checks, and scheduling are three separate systems, and a URL can die at any of them. The usual suspects are a missing or malformed href, a robots rule on the source or destination, a redirect chain or bad status code, a link that only exists after JavaScript runs, or a scheduler that deduplicates or deprioritizes the URL before it ever hits the fetch queue.

TL;DR:

  • Extracted URLs can still be blocked before crawling due to robots.txt rules, meta tags, HTTP status codes, or scheduler filters, regardless of successful extraction.
  • JavaScript-only links require rendering with tools like Puppeteer to be discovered, and delays in rendering can impact crawl timing and coverage.
  • Proper diagnosis involves checking raw HTML for the href, verifying robots.txt permissions, inspecting redirect chains, and comparing raw versus rendered DOM for JS-driven links.
  • Googlebot relies on server-visible anchors for initial discovery and treats JavaScript links as secondary, while Nutch extracts outlinks during parsing based on configuration, affecting how missing links are diagnosed.
  • Gyrence improves failure diagnostics by categorizing errors such as extraction failure, robots blocking, or redirect issues as separate, actionable signals within the crawl process.
Gyrence
Debug Crawls With Clearer Signals
Gyrence separates extraction, robots, redirects, and other failure cases so developers can reason about outlink crawling results.
Explore Gyrence

Table of Contents

Extraction is the parsing step where a crawler reads a fetched page and pulls out candidate URLs. Scheduling decides which of those candidates get queued. Fetching is the HTTP request that actually retrieves them. Confusing these three is the single biggest source of wasted debugging time.

Crawlers reliably discover links from <a> elements carrying an href attribute. That's the contract Google documents for crawlable links, and Apache Nutch's parser follows the same principle at the HTML level.

Constructs that routinely get ignored:

  • onclick-only navigation with no href fallback
  • Links built entirely from data-* attributes and reassembled by a script
  • Fragment identifiers appended to an otherwise valid path (they resolve, but they're treated as the same document, not a new URL)
  • Malformed relative paths that fail to resolve against the base URL

A crawler normalizes each extracted href against the page's base URL, converting relative paths to absolute ones and stripping default ports, trailing dots, and duplicate slashes. A rel="nofollow" or rel="sponsored" attribute doesn't block extraction. It's a hint about how the destination should be weighted or followed, and different crawlers apply that hint differently.

Once a URL survives extraction, several independent checkpoints can still stop it before a fetch happens. Work through these in order when a link you can see in the raw HTML never shows up in your crawl output:

  1. Robots.txt on the destination host. If the target disallows the path, the crawler won't request it, full stop. Nutch caches these rules per host, protocol, and port, and follows a configurable number of redirects when fetching robots.txt itself, so a stale cache can block a URL long after the source file changed.
  2. Robots meta tags and X-Robots-Tag headers. A page can be perfectly linkable and still carry a noindex, nofollow directive at the HTML or HTTP header level. Google's documentation is explicit that a page disallowed by robots.txt never gets fetched in the first place, which means any meta directive on that page is irrelevant. Crawlers can't read a header on a page they never requested.
  3. Rel attribute semantics. nofollow, sponsored, and ugc no longer guarantee a link is skipped entirely. Modern crawlers treat them as one signal among many, not a hard block.
  4. Status codes and redirect chains. A 404, a 500, or a redirect loop kills the fetch. Long redirect chains often get truncated before reaching the final destination.
  5. JavaScript-inserted links. If the anchor only exists in the rendered DOM, the crawler must render the page first. Rendering runs on a separate, delayed queue.
  6. Scheduler-side filtering. Deduplication, per-host concurrency caps, and crawl-budget heuristics can silently drop a URL that was successfully parsed and queued.
  7. Pipeline filters after parsing. In Nutch, an IndexingFilter or URLExemptionFilter can remove or rewrite outlinks after extraction. A URL that shows up in the parser's output can still vanish before it reaches the LinkDb.

Pro Tip: Don't stop debugging at the parser. Production incidents involving Nutch pipelines often trace back to a downstream IndexingFilter silently stripping links the parser extracted correctly. Check the filter chain, not just the parse log.

Both systems separate extraction from scheduling, but they diverge on where and when rendering happens, and on how much you can configure.

Googlebot crawls mobile-first and renders JavaScript with headless Chromium as a second, delayed pass after the initial HTML fetch. Links that only appear post-render get queued for discovery, but that discovery can lag the raw HTML crawl by hours or days. Google's own guidance treats server-visible anchors as the primary discovery mechanism and JavaScript navigation as something you verify separately, never assume.

Parallel HTML and rendered link discovery paths

Apache Nutch extracts outlinks at parse time, using the OutlinkExtractor to pull links directly out of the fetched document. Nutch stores the results in a LinkDb, an inverted map of incoming links built by inverting the outlink data. You can configure Nutch to ignore external links entirely or store only the host portion of a URL, which means "parsed successfully" in a Nutch log can still mean the full link never made it into a fetch queue.

The practical difference for debugging:

  • With Googlebot, trust the rendered DOM over raw HTML whenever JavaScript is involved. Raw HTML tells you what's discoverable immediately; the rendered snapshot tells you what's discoverable eventually.
  • With Nutch, trust your own configuration over the parser log. If outlinks aren't showing up downstream, check ignore.external.links and your IndexingFilter chain before assuming a bug.

Run through this checklist in order. Each step isolates one failure stage, so stop as soon as you find the break.

  1. Confirm the href exists in raw HTML. Run curl or view source on the page. If the link isn't in a proper <a href>, extraction never had a chance.
  2. Check robots.txt for both hosts. Fetch robots.txt on the source and destination directly and look for disallow rules covering the path.
  3. Inspect rel attributes, meta robots tags, and X-Robots-Tag headers. Use curl -I to see response headers, since meta tags and headers can both apply directives.
  4. Trace the full HTTP chain. Run curl -I -L and count redirects. A chain longer than four or five hops is a common silent killer.
  5. Render the page and diff the DOM. Use Puppeteer or Playwright to capture the rendered HTML and compare it against the raw fetch. If rendering surfaces links the raw HTML lacks, you've found a JS-discovery gap.
  6. Review your crawler's queue and LinkDb output. Check whether the URL was scheduled, deduplicated, or dropped by a filter before it ever reached the fetcher.
  7. Reproduce in a minimal sandbox. Strip the crawler down to plain HTML extraction, then add rendering back in. This single test usually tells you whether the problem is the site's markup or your own pipeline.

A useful reference point: robots rules in Nutch are cached per host, protocol, and port, so a robots.txt fetch that failed hours ago can keep blocking discovery until the cache expires, even after the site fixes the file.

Gyrence's five primitives, Search, Traverse, Fetch, Extract, and Map, each return a typed, discriminated-union response. That matters here specifically: a failed fetch, a blocked robots rule, and a redirect loop all come back as distinct, labeled error types instead of one generic failure, so you know which stage broke without re-running the whole crawl.

One internal case study, documented after an agent got stuck looping on Wikipedia pagination, traced the root cause to pagination links the agent kept rediscovering without a stopping condition. The fix combined depth caps with dedup keyed on the normalized URL, not the raw one.

Safe defaults worth adopting regardless of stack:

  • Cap crawl depth conservatively and revisit the number only with data behind it.
  • Limit per-host concurrency so one slow domain doesn't starve the rest of the queue.
  • Surface failure types (extraction, policy, transport) as distinct fields, never one flat error string.

Pro Tip: When a link graph gets tangled, map it before you crawl it. Typed link graph mapping shows you the shape of a domain's outlinks before you spend fetch budget walking it blind.

What Should Developers Actually Check First?

Href-first discovery is the contract, not a suggestion. If a link isn't in a server-rendered anchor, treat its discovery as optional, not guaranteed. In under ten minutes: pull the raw HTML, check robots.txt on both ends, and diff the rendered DOM against the raw fetch. Whatever your crawler's logs don't label explicitly, assume it's hiding a bug.

— Glen

Most crawling tools tell you a fetch failed. Gyrence tells you why, extraction miss, robots block, redirect loop, or timeout, as a typed field you can branch on in code instead of parsing an error string. That's the practical difference for anyone who's spent an afternoon guessing which of five failure modes actually killed a crawl.

Gyrence

Start with a small traversal using Gyrence's Traverse primitive on a domain you already know has tricky pagination or JavaScript-rendered navigation, and watch how the response distinguishes an extraction miss from a policy block. If you're running crawls that need to catch fetch failures on a schedule rather than after the fact, WebDoppler adds webhook alerts when outlink resolution breaks. Plans start at the Standard tier at $75 per month, with a free tier and pay-as-you-go option available for testing before you commit to a workspace.

FAQ

The link was extracted but blocked or skipped at a later stage, usually a robots.txt disallow rule, a redirect that fails partway through the chain, or a scheduler-side dedup decision. Extraction and fetching are separate steps, and Google's documentation on crawlable links confirms that a valid href only guarantees discovery, not a fetch.

Not necessarily. Modern crawlers treat nofollow, sponsored, and ugc as signals about how to weight or follow a link, not an absolute block on fetching it. A robots.txt disallow rule or a robots meta directive has a much stronger, more consistent effect.

Render the page with a headless browser like Puppeteer or Playwright and compare the resulting DOM against the raw HTML fetch. If the anchor appears only in the rendered version, it depends on client-side execution that a plain HTML crawler will never see.

Gyrence's Fetch and Traverse primitives normalize and clean pages before extraction, and failures surface as typed responses rather than silent empty results. Details on which content types and rendering paths are supported are available through the Gyrence API documentation.

Outlink processing extracts URLs a page points to during parsing; inlink processing, like Nutch's LinkDb, inverts that data to build a map of which pages point to a given URL. Outlinks drive what gets crawled next; inlinks are typically used for ranking signals and link graph analysis after the crawl completes.