Visit Website →
← Back to blog

Engineers: Deploy 512 Token RAG Chunking in a Two Week Pilot

September 15, 2026
Engineers: Deploy 512 Token RAG Chunking in a Two Week Pilot

For most web content, the right default is recursive character splitting at roughly 512 tokens with 10 to 20 percent overlap, respecting document headings above all else. Reach for contextual retrieval only on small, accuracy-critical corpora, late chunking for long reference documents, and semantic chunking once you already have an eval set to justify it. Build a 30 to 100 query labeled eval set and measure recall@k and MRR before you trust any of it.

TL;DR:

  • The recommended default chunk size for most web content is around 512 tokens with 10 to 20 percent overlap, adjusted based on content density.
  • Fixed-size and recursive splitters are the most cost-effective, while semantic and hierarchical chunking offer limited consistent benefits.
  • For HTML pages, structural parsing to identify main content, headings, and handle tables and code blocks separately improves chunk quality.
  • Storing both small chunks with metadata and entire parent sections enables more accurate retrieval and reduced hallucinations.
  • Building a labeled eval set from real queries and measuring recall@k and MRR ensures your chunking strategy effectively balances cost and retrieval accuracy.
Gyrence
Build Cleaner RAG Inputs
Gyrence turns web pages into structured, agent-ready data with markdown fetching and predictable, typed responses for your ingestion pipeline.
Explore Gyrence

Table of Contents

Comparing Chunking Strategies for RAG

Every chunking approach trades index-time cost against retrieval accuracy, and the trade curve is steeper than most teams expect. Fixed-size and recursive splitters cut text at character or token boundaries, falling back through paragraph, sentence, and word breaks until a chunk fits its budget. They're cheap, deterministic, and the closest thing this field has to a safe default, which is why Microsoft's Azure architecture guidance treats them as the starting point before testing anything more elaborate.

Sentence and paragraph chunking respects natural language boundaries but produces uneven chunk sizes, which complicates batching and embedding costs. Semantic chunking clusters sentences by embedding similarity, aiming to keep single ideas together. It sounds smarter than it performs: a systematic evaluation on arXiv found that expensive chunking methods, semantic chunking included, often deliver little consistent benefit over a well-tuned fixed-size baseline.

Hierarchical or parent-child chunking indexes small chunks for retrieval but stores larger parent sections for context assembly. Late chunking embeds the full document first, then splits, so each chunk's vector still carries surrounding context. It needs a long-context embedding model and adds compute at index time. Contextual retrieval, popularized by Anthropic's research, uses an LLM to prepend a short contextual summary to each chunk before embedding. It fixes the "lost context" problem directly but adds one LLM call per chunk, which gets expensive fast on large corpora.

Rules of thumb that hold up in practice:

  • Start with recursive/fixed-size chunking. Almost nothing beats it on cost.
  • Reach for late chunking on long reference docs where losing surrounding context hurts recall.
  • Use contextual retrieval only on bounded, high-stakes corpora where accuracy justifies the LLM spend.
  • Treat semantic chunking as a hypothesis to test, not a default. Verify it against your own eval set first.

How Do You Chunk HTML Web Pages Correctly?

Raw HTML is noisy. Nav bars, footers, cookie banners, and sidebar widgets all compete for space with the content you actually want, and a naive character splitter will happily chunk a "subscribe to our newsletter" block right alongside your API documentation. The fix starts with structural parsing, not character counting.

  1. Identify the main content node. Strip <nav>, <footer>, <aside>, and known ad/widget containers before you touch chunking logic.
  2. Preserve the heading path. Track each chunk's position in the document tree (H1 > H2 > H3) so a chunk about "Refund Policy > Exceptions" doesn't lose that context when isolated.
  3. Split on headings and sections first, sentences second. LLMBestPractices recommends splitting on structural boundaries before enforcing token budgets, then falling back to sentence or token splits within an oversized section.
  4. Handle tables and images separately. An IBM cookbook on RAG chunking suggests using an LLM to summarize or caption non-text elements before folding them into the surrounding chunk, rather than splitting a table mid-row.
  5. Keep code blocks intact. Never split a code fence across two chunks. Treat it as an atomic unit even if it exceeds your normal token budget.
  6. Store a parent pointer. Every chunk should reference the section or page it came from, so you can reassemble context or cite the source page accurately.

What Chunk Size and Overlap Should You Use?

A commonly recommended baseline is around 512 tokens with 10 to 20 percent overlap. This balance tends to keep chunks small enough for precise retrieval but large enough to hold a complete thought. Production RAG guidance treats this configuration as the practical starting point most teams should test first.

  • Safe range: 256 to 1,024 tokens, adjusted by document type. Dense reference material often does better tighter; narrative content can run longer.
  • Avoid sub-200 token chunks for factual, dense content. You'll fragment answers across too many chunks and hurt recall.
  • Avoid chunks over 800 tokens that blend multiple topics. Retrieval precision drops when a chunk answers three different questions at once.
  • Measure with the actual tokenizer of your target embedding or LLM model, not an estimate. Token counts vary meaningfully between tokenizers.

Semantic chunking and contextual retrieval each add embedding or LLM passes at index time, and that cost compounds across large corpora, so re-verify chunk size whenever you switch embedding models.

How Should You Structure Metadata for Chunks?

Every chunk needs enough metadata to be traceable back to its source and reassembled with context at query time. The required fields are source_url, title, heading_path, chunk_index, and last_updated. Optional fields like lang, author, tenant_id, and tags help with filtering in multi-tenant or multilingual systems.

The parent-child pattern solves a real tension: small chunks retrieve accurately, but large chunks answer questions better. The fix is to store both.

  1. Embed and index small child chunks (the 512-token units) for retrieval.
  2. Store the full parent section (the entire heading-level block) separately, keyed by a shared ID.
  3. At query time, retrieve on the child chunk, then resolve and return the parent section for generation, citing the parent's source_url and heading_path.

This "store small, retrieve small, render large" approach shows up across most production RAG ingestion architectures, and it's one of the cheapest fixes available for reducing hallucinations caused by truncated context.

How Do You Evaluate Chunking Choices on Your Own Corpus?

Guessing which chunking strategy works best is a waste of engineering time you don't have. Build a labeled eval set of real queries mapped to their ground-truth passages from your actual corpus, not a generic benchmark.

  • Measure recall@k: does the correct passage appear in your top k retrieved chunks?
  • Measure MRR (mean reciprocal rank): how high does the correct passage rank when it does appear?
  • Track downstream answer quality and hallucination rate, since retrieval accuracy doesn't always translate one-to-one into correct generated answers.
  • Log embedding and LLM call counts per ingestion run, so cost increases are visible before they hit a bill.

The recommended sequence, echoed across practitioner writeups on chunking strategy, is to baseline with a recursive splitter, add hybrid search or a reranker, and only test contextual retrieval or late chunking if the eval numbers justify the added cost.

Pro Tip: Rerun your eval set every time you change embedding models, not just when you change chunking logic. A model swap can silently shift which chunk size performs best, and skipping this check is how teams end up debugging "mystery" recall drops weeks later.

What I've Learned Running Chunking Pilots

What I've Learned Running Chunking Pilots — overview diagram

Most teams over-invest in chunking sophistication before they've measured anything. A two-week pilot beats months of tuning: ingest a representative slice of your corpus, run the 512-token recursive baseline, build your eval set, and measure cost against recall before touching anything fancier.

Semantic similarity thresholds are more fragile than vendors let on. They drift across domains and content types, which is exactly why an evaluation-first workflow beats intuition. Version your chunker configuration in your metadata. When a page changes and you re-chunk it, you need to know which chunking logic produced which vectors, or you'll spend a debugging session chasing a phantom bug.

— Glen

Where Gyrence Fits Into Your Chunking Pipeline

Most chunking failures aren't logic bugs. They're upstream parsing problems: a nav bar that leaks into your main content extraction, a heading tag that got flattened during scraping, a table that arrived as a wall of unstructured text. Built to eliminate that class of failure before your chunker ever runs. A fetch-to-markdown pipeline preserves heading structure automatically, so heading_path metadata stays reliable without custom parsing glue. A schema-guided JSON extraction process handles structured fields, and responses come as typed, discriminated unions, so ingestion pipelines can distinguish between cleanly fetched pages and partial failures instead of silently chunking garbage.

Gyrence

Spending controls help keep index-time costs predictable even when running experimental strategies like contextual retrieval across a large crawl. If your current scraping setup keeps handing your chunker malformed markdown, check the Gyrence API and see whether cleaner input fixes more of your recall problem than a fancier chunking strategy ever could.

Sources

FAQ

What Is the Best Default Chunk Size for RAG?

Around 512 tokens with 10 to 20 percent overlap works as a reliable starting point for most web content, based on production RAG guidance. Adjust within the 256 to 1,024 token range depending on how dense or narrative your content is.

Is Semantic Chunking Worth the Extra Cost?

Usually not as a starting point. A systematic evaluation found that semantic chunking often underperforms a well-tuned fixed-size baseline, so test it against your own eval set before adopting it.

How Do I Chunk Tables and Images on a Web Page?

Extract them separately and use an LLM to generate a caption or summary before folding that text into the surrounding chunk, rather than splitting a table mid-row, following the approach in IBM's RAG chunking cookbook.

What Metadata Should Every Chunk Include?

At minimum, attach source_url, title, heading_path, chunk_index, and last_updated. This lets you trace, reassemble, and correctly cite any chunk returned during retrieval.

How Do I Know If My Chunking Strategy Is Actually Working?

Build a 30 to 100 query labeled eval set from real queries against your corpus and measure recall@k and MRR. Tools that produce clean, heading-aware markdown, like a structured fetch pipeline, reduce the parsing noise that otherwise skews these measurements.