Boilerplate removal is the step that strips a page's repeated surroundings (navigation, footers, ads, consent notices, related-article panels) and keeps the main content. Every method does this by guessing, so every method has failure modes. Below: what gets removed, how the main approaches decide, what they wrongly discard, and how to test one before it feeds a RAG or LLM pipeline.
TL;DR
- Boilerplate is the repeated page chrome around the content: navigation, legal text, headers, footers, ads, share widgets.
- Methods fall into three groups: rule-based cascades, tree and text-density heuristics, and trained block classifiers. None is right on every page.
- Typical failures are discarded tables, quotes, summaries and forum posts. Over-cleaning hurts retrieval as much as under-cleaning.
- Keep the original page and its metadata next to the cleaned text, and judge cleanup by final answer quality, not characters removed.
Table of Contents
- What boilerplate removal actually removes
- How tools tell content from clutter
- What the numbers say
- What boilerplate removal gets wrong
- A perspective from Gyrence
- How to evaluate and integrate it in a pipeline
- Skip the extractor maintenance
- FAQ
- Sources
What boilerplate removal actually removes
An annotation guideline from the CLEANEVAL shared task lists the material to strip: navigation, link lists, copyright and legal information, repeated headers and footers, templates, ads and automated blog spam. The Trafilatura evaluation frames the same task as keeping the main content while discarding headers, footers, sidebars, author and footer information, and social links.
For scraping, this matters for a plain reason. If a footer link list ends up in your index, chunks can match queries for irrelevant reasons. UnRAG's ingestion guidance names navigation, cookie banners, footer links and repeated headers as sources of that retrieval pollution.
How tools tell content from clutter
There is no tag that says "this is the article." Extractors infer it from several signals, which the Trafilatura paper describes:
- Tree-based heuristics: the HTML structure and a node's ancestors.
- Shallow text features: word length, text-to-markup ratio, punctuation ratio.
- Position: where a block sits locally and globally on the page.
- Inter-document frequency: blocks that recur across many pages of a site are probably template.
- Machine learning: sequence-labeling models that classify blocks.
The weakness is that the signals depend on authors writing sane markup. Researchers at Bauhaus-Universität Weimar note that after two decades of work the problem stays hard, partly because authors often misuse semantic HTML5 elements.
Rule-based versus learned
Rule-based cascades use HTML elements and content heuristics. Machine-learning methods classify blocks. The Trafilatura authors observe that rule-based approaches are easier to use and parameterize. Some cleaners go simpler still: a curated list of CSS selectors that trims headers, footers, navigation, sidebars, ads, cookie bars and share widgets, as in this open-source cleaner. It is fast and predictable, and it misses anything not on the list.
Older tree-based tools such as Boilerpipe use decision trees over HTML features. Tree analysis works well on narrow document classes: researchers at IPN report that content on modern sites forms groups in the tree, and their HTML-tree filter raised Boilerpipe's precision on news articles by at least 15%.
What the numbers say
The 2021 ACL paper reports Trafilatura 0.8.2 at precision 0.934, recall 0.890, accuracy 0.914 and F-score 0.912 on its benchmark. The project's current evaluation page reports F-scores of 0.919 to 0.926 for version 2.3.0, notes that some extractors are noticeably slower, and finds no single extractor best across all datasets and complexity levels.
Read those as benchmark results, not production guarantees. The same page describes the precision and recall trade-off: presets exist for high-precision clean text or high-recall corpus building. A generic approach saves time and compute at some cost in accuracy, so you choose where to sit.
What boilerplate removal gets wrong
This is the part most guides skip. The IPN study found that the method wrongly discards summaries and quotes, and that Boilerpipe frequently drops tables. It also struggles with forums, mailing lists, blog covers and section pages, where relevant content is spread across distant branches of the tree.
Structured documents are riskier. OCRByte warns that indiscriminately deleting repeated content in contracts or reports can destroy meaning, such as section identity or page order. UnRAG's rule of thumb is to keep headings, lists, code blocks, warnings, notes, examples, FAQ labels and freshness or scope metadata whenever they change meaning. Do not remove anything whose loss changes usefulness to a reader.
That is why we keep headings and structure in normalized output. For more on this trade-off, see our piece on preserving HTML in content normalization.
A perspective from Gyrence
We think boilerplate removal is usually framed as a deletion problem, and it should be framed as a reporting problem. A cleaner that silently returns an empty string for a page it misread is worse than one that says it is unsure.
That is how we build the Fetch primitive: a typed, discriminated-union response where failure cases are first-class, so an agent can branch on "no main content found" instead of ingesting a cookie banner as if it were the article. Web data is a storm; a pipeline that hides the weather is the one that surprises you later.
Our practical advice: pick the least aggressive cleaning that keeps your retrieval quality, keep the raw source, and treat every extractor as a hypothesis you test against your own pages.
— Glen
How to evaluate and integrate it in a pipeline
- Build a test set. The Trafilatura evaluation used 500 documents, mixing typical articles with harder pages that have mixed content or malformed HTML. Do the same with your own target domains.
- Score it. Compare precision, recall, accuracy and F-score per extractor, per page type.
- Audit samples by eye. Check that headings and table context survived before anything reaches an index.
- Store provenance. BuildRAG suggests keeping the original URL, title, canonical URL when present, retrieval time and headings, plus a content hash to detect changes.
- Canonicalize URLs so the same page is not ingested twice.
- Keep both versions. A 2026 preprocessing guide recommends vectorizing the cleaned chunk while preserving the original text and source metadata for auditability.
- Measure the end result. OCRByte's advice is to judge cleanup by final answer quality, not characters removed or index size saved.
If you want a quick outside view of how an LLM will read a cleaned page, an LLM readability checker is a cheap sanity check on the output.
For the wider pipeline, our guides on RAG web ingestion and fetching pages to markdown for LLMs cover the steps before and after cleaning.
Skip the extractor maintenance
Gyrence's Fetch returns a cleaned page as markdown, and Extract returns schema-guided JSON, both with typed failure responses. Spending caps keep the bill predictable, and there is no separate AI extraction charge. Plans run from Free through Standard at $75 per month, Growth at $299 and Scale at $549, plus Pay-As-You-Go; see pricing. If you need to know when a source page changes, WebDoppler monitors pages with webhook alerts.
FAQ
Is boilerplate removal the same as HTML cleaning?
Not quite. HTML cleaning strips code such as scripts and markup. Boilerplate removal also drops human-readable text that is not the main content, like menus, legal notices and ad blocks. Most pipelines do both.
Should I use rules or machine learning?
Rules are simpler to run and tune, and they are predictable. Learned classifiers can handle more layouts but need training data and testing. Many mature tools combine a rule cascade with fallbacks, so test on your own pages before choosing.
Can boilerplate removal delete useful content?
Yes. Published studies show tables, quotes, summaries and forum posts being dropped. Contracts and reports are especially sensitive because repeated headers can carry section or page identity. Audit samples and keep the original page.
How do I know my cleanup is good enough?
Score precision and recall on a hand-labeled set of your own pages, then check downstream answer quality. Removed-character counts say little about whether retrieval improved.
Sources
- CLEANEVAL: Guidelines for annotators
- Trafilatura: A Web Scraping Library and Command-Line Tool
- Trafilatura benchmarks and evaluation
- An Empirical Comparison of Web Content Extraction Algorithms
- Improving the Boilerpipe Algorithm for Boilerplate Removal
- html_clean.rs source
- UnRAG: Cleaning and normalization
- OCRByte: Boilerplate Removal for OCR and LLM Pipelines
- BuildRAG: Parse webpages without indexing the menu
- RAG Data Preprocessing: Ingestion, Chunking & Metadata