← Back to blog

Page Diff Algorithms: How to Detect Real Changes on the Web

November 22, 2026
Page Diff Algorithms: How to Detect Real Changes on the Web

A page diff algorithm compares two saved versions of a web page and reports what changed. The hard part is not finding differences. It is separating real changes (a price, a clause, a headline) from noise (ad slots, timestamps, session tokens). The practical answer: normalize the page first, diff a structured representation, and keep every change tied to its source URL and fetch time. Gyrence's Fetch and Extract primitives return cleaned markdown and typed JSON, which gives a diff stable input to work with, and WebDoppler monitors pages and sends webhook alerts.

Gyrence
Track meaningful page changes
Gyrence’s Fetch and Extract primitives provide cleaned markdown and typed JSON for comparing web pages with more structured input.
Visit

TL;DR

  • Text diffs (LCS, Myers) are fast but fire on every cosmetic change. DOM-aware diffs understand structure but are costlier.
  • Normalization decides your false-alarm rate more than the algorithm does: strip whitespace, comments, scripts and volatile fields before comparing.
  • Hashing a normalized page tells you whether it changed. Structured diffs tell you what changed.
  • For RAG and compliance, store each version with its URL and timestamp so every change is traceable.
  • Schedule checks by how often each page actually changes, not on one global timer.

Table of Contents

What a page diff compares

A classic text diff compares two sequences of lines and finds the shortest set of insertions and deletions turning one into the other. A web page is not a flat sequence. It is a tree of HTML elements, and the same visible content can sit in different markup after a template change.

According to a 2020 ACM Computing Surveys review of web page change detection, most systems convert saved versions into an XML-like form where each element represents opening and closing tags, then compute similarity against the latest crawl. Many let users set a threshold and notify only when change exceeds it. That threshold is your first noise filter.

The main algorithm families

Oita et al. catalog the approaches. In practice they fall into four groups.

FamilyHow it worksGood forWeak spot
String / sequenceJaccard word overlap, k-shingling, longest common subsequence (as in GNU diff), edit scriptsArticle text, markdown outputReordered blocks look like delete plus insert
Tree / DOMMatch nodes between two DOM trees, emit insert/delete/change operationsStructural changes, layout shiftsCost grows with node count
Graph matchingBipartite matching with the Hungarian algorithm (CMW, MH-Diff)Moved subtreesExpensive on large pages
Page DigestEnumerate nodes depth-first and encode content to preserve text order (SDiff, RMS)Bulk monitoringLess intuitive output

The same paper reports that the Page Digest algorithms run in linear time without heavy heuristics, while still capturing a wide range of changes.

A DOM-aware example

VWO's engineering team documents a DOM comparator that strips whitespace and comment nodes, matches nodes between two trees with a 0 to 1 similarity score (above 0.5 counts as a good match), and outputs an operation list. Unmatched original nodes are deletions; unmatched new nodes are insertions. That operation list is what you want downstream: it is machine-readable.

Near-duplicate detection at scale

When you track millions of pages, you first need a cheap "did anything meaningful change?" test. Simhash fingerprints serve this purpose. Google's work on near-duplicates, as summarized by Shifter's engineering blog, found that for a repository of 8 billion pages, 64-bit fingerprints with at most three differing bits were a reasonable near-duplicate threshold. Use the hash as a gate, then run the expensive diff only on pages that pass it.

Accuracy and speed: what the benchmarks say

The choice is a tradeoff, and the numbers show it:

  • The WebEvo paper reports a plain DOM-tree detector at precision 0.40, recall 0.73 and F1 0.50. Its DOM-plus-semantic-visual approach reached F1 0.84, and ran 42.01% faster than a whole-page visual tool.
  • A 2008 DKE study found that restricting similarity computation to nodes with the same tag, with hashed direct node access, was at least 30% faster on pages over 500 nodes. Pages near 1,000 nodes took about 8 seconds.

Two lessons. A naive DOM diff produces many false positives (low precision), so structure alone is not enough. And node-to-node similarity gets slow on big pages, so constrain what you compare.

Normalize before you diff

Most false alerts come from input, not from the algorithm. A workable normalization pass:

  1. Reduce to content. Convert the page to markdown or extract only the fields you care about. Navigation, footers, cookie banners and ad slots drop out.
  2. Strip volatile tokens. Remove timestamps, CSRF tokens, session IDs, randomized class names and tracking parameters.
  3. Canonicalize. Collapse whitespace, normalize URLs and sort attributes.
  4. Hash. Store a SHA-256 of the normalized result for a cheap equality check.
  5. Diff only on a hash mismatch, and apply a change threshold before alerting.

Normalization quality bounds everything after it. An article-body extraction evaluation found that even the best extractor scored only 0.470 exact-match accuracy after normalization. Benchmark your extraction before you trust the diff on top of it.

A field-level diff is often cleaner than a page-level one. If you track a competitor's pricing, extract {plan, price, currency} as JSON and compare the fields. A redesign then changes nothing, because the data did not change. We cover that scenario in planning for a pricing page redesign.

Keeping diffs traceable for RAG and compliance

For AI agents and financial or compliance research, "the page changed" is not enough. You need to prove which page, when, and what.

The EDGI web-monitoring architecture is a good vocabulary. A Version is a snapshot of a page at a point in time. A Change is two different versions of the same page. A Diff is the representation of that change, as plain text or rendered HTML. Annotations, human or automated, characterize each change for audit.

For RAG ingestion, the DocForge roadmap shows a clean pattern: normalize each page to markdown, hash it, diff runs into new, changed, deleted and unchanged, and gate deletions on crawl success. Every vector chunk carries its source URL, so a changed page becomes a targeted delete and re-embed instead of a full rebuild.

Page snapshots become traceable targeted vector updates

The gate on deletions matters. A failed fetch is not a deleted page. Gyrence returns typed, discriminated-union responses that include failure cases, so your pipeline can tell "page gone" from "request blocked" instead of guessing.

Schedule by change rate

The 2020 survey notes that systems need to estimate each page's change frequency to build efficient checking schedules and cut wasted checks. The 2008 study, using 520 CNN.com homepage snapshots from the Wayback Machine, found the fraction of changed pages decayed roughly exponentially, like a Poisson process. Practically: check fast-moving pages often, static pages rarely, and adapt as you learn. See our notes on priority URL scheduling and on how crawl-rate planning shapes monitoring. Conditional requests help too; our guide to HTTP caching for scrapers covers cutting bandwidth on unchanged pages.

Promo: monitoring without building the diff stack

WebDoppler monitors a page and sends webhook alerts, with structured data capture. Gyrence bills by credit with spending caps, and AI extraction carries no separate charge, so a monitoring job does not produce a surprise invoice. The Standard plan is $75 per month, Growth is $299 per month and Scale is $549 per month; there is also a Free plan and Pay-As-You-Go. See the pricing page.

Gyrence

Our perspective

Most diff problems are extraction problems wearing a disguise. Teams pick Myers or a tree matcher, then drown in alerts because the input still contains banners, timestamps and rotating IDs.

We build Gyrence around that. Fetch returns cleaned markdown, Extract returns typed JSON against your schema, and every response, failures included, is typed. When a fetch fails, your system knows it failed and does not record a deletion.

Diff the data you care about, keep the source URL attached, and let failure modes be visible. That is what traceable change detection looks like.

Glen

FAQ

Which page diff algorithm should I start with?

Normalize to markdown, hash with SHA-256, and run a line or word-level diff only when hashes differ. Add a DOM-aware diff when structure itself is the signal, such as layout or form changes.

How do I avoid false change alerts?

Strip scripts, comments, timestamps and tokens before comparing, and set a change threshold so trivial edits do not notify. Comparing extracted fields instead of whole pages removes most remaining noise.

Is a visual diff better than a DOM diff?

They catch different things. In the WebEvo results, combining DOM, semantic and visual signals reached F1 0.84 against 0.50 for a DOM-only detector, so mixing signals improves accuracy.

How do I keep changes traceable?

Store each version with its URL, fetch time and content hash, and link every diff to the two versions it compares. Carry the source URL on every chunk you embed.

Sources