Vector database ingestion is the pipeline that turns source content into searchable embeddings: fetch, clean, chunk, embed, store with metadata, then keep it fresh. For web data, most failures happen before the embedding step, at fetch and extraction. If a page is fetched badly or cleaned badly, retrieval can't recover it. The fix is to treat the front of the pipeline as typed, observable and cost-capped. That is what we built Gyrence to do.
TL;DR
- Ingestion is five stages: fetch and clean, chunk, embed, upsert with metadata, refresh. Quality lost early is never recovered later.
- Chunk by structure (headings, paragraphs, tables), not by raw character count, and attach URL, fetch time and content hash to every chunk.
- Use deterministic chunk IDs and content hashes so re-runs are idempotent and only changed pages get re-embedded.
- Make failures first-class: a blocked page, a 404 and an empty extraction should be distinct results, not silent gaps in your index.
- Cap spend per run. Cost per ingested record matters more than raw throughput.
Table of Contents
- What ingestion actually involves
- Stage 1: Fetch and clean the page
- Stage 2: Chunk by structure
- Stage 3: Embed in batches
- Stage 4: Upsert with provenance and stable IDs
- Stage 5: Refresh without re-embedding everything
- Controlling the bill
- Verify before you trust the index
- Our perspective
- Start ingesting with Gyrence
- FAQ
- Sources
What ingestion actually involves
According to Anyscale's RAG documentation, ingestion loads documents (PDF, DOCX, PPTX, HTML, TXT), applies a chunking strategy, embeds the chunks and stores them in a vector database. At query time the user's request is encoded to a vector and matched against those embeddings. The top-ranked chunks and their metadata become context for an answer that can cite its sources.
Microsoft's example RAG ingestion pipeline shows the same shape for web pages: download the page, extract text, split it into chunks small enough for the LLM, generate an embedding per chunk, create a collection, and store text, vectors and metadata such as source ID, title and source URL. Its default store is Postgres with pgvector, with Qdrant and Azure AI Search as alternatives.

The vector database is the easy part. The hard part is everything that feeds it.
Stage 1: Fetch and clean the page
Web data is noisy: navigation, cookie banners, footers, scripts and duplicated boilerplate. If you embed that, your nearest-neighbor results match on boilerplate instead of content. Anyscale's guidance is blunt on this point: high-quality preprocessing is critical because retrieval can't recover information that was never extracted properly.
What the fetch stage should do:
- Respect access rules. Read
robots.txtbefore the first request. The Robots Exclusion Protocol is standardized as RFC 9309, including how clients should handle errors and caching. Our guide to robots meta tags and scraping covers the page-level signals a crawler should also read. - Return clean markdown. Markdown keeps headings, lists and tables, which are exactly the boundaries your chunker wants.
- Surface failures by type. A timeout, a block, a redirect and a 404 each call for a different action. A pipeline that collapses them into "empty string" indexes nothing and tells you nothing.
- Preserve structure. Stripping too aggressively loses tables and code blocks. See why we preserve HTML during content normalization.
The Gyrence Fetch primitive fetches a page and cleans it to markdown. Every call returns a typed, discriminated-union response, failure cases included, so your code branches on the result instead of guessing. For discovery, Map returns a domain's URL graph via sitemaps, Traverse crawls outward from a starting URL, and Search finds starting points. Together they cover "what pages exist" before you spend anything embedding them.
Stage 2: Chunk by structure
Anyscale suggests typical chunk sizes of 300 to 500 tokens with 10 to 20 percent overlap, preferring semantic or structural splits (section, paragraph, table) over arbitrary cuts. With markdown input, this is straightforward: split at headings, keep the heading path with the chunk, and split oversized sections at paragraph boundaries. Count tokens with your embedding model's own tokenizer, not character estimates.
A chunk with no context retrieves badly. "Set it to 30" means nothing without the heading above it. Carrying the section path as metadata lets you prepend it at embed time or show it in a citation.

Stage 3: Embed in batches
Embedding is network-bound, so batch requests and run them concurrently up to your provider's rate limit. Dimension is a storage lever as well: OpenAI's embeddings guide documents a dimensions parameter on its third-generation models that returns shorter vectors. Whatever model you pick, record the model name and version on every chunk. When you change models, old and new vectors are not comparable, and you will need to rebuild the collection.
Stage 4: Upsert with provenance and stable IDs
Store the vector next to the metadata that makes the answer trustworthy. At minimum:
| Field | Why it matters |
|---|---|
url | Citation and per-page deletion |
fetched_at | Freshness filtering and audits |
title, section_path | Readable citations, context at embed time |
content_hash | Skip unchanged chunks on re-runs |
token_count | Budget checks |
embedding_model | Safe re-indexing when the model changes |
Derive each chunk ID deterministically, for example a hash of the URL, chunk index and chunk text. Re-running the job then upserts the same rows instead of duplicating them, and a changed page produces new IDs while the old chunks for that URL can be deleted.
If you run Postgres, pgvector keeps vectors in the same database as your metadata and supports HNSW and IVFFlat indexes, which means one transaction can replace a page's chunks atomically.
For compliance and financial-research work, provenance is the product. Every answer should trace to a URL and a fetch timestamp. Our write-up on audit-ready controls for compliance-friendly scraping lists what to log.
Stage 5: Refresh without re-embedding everything
A stale index makes an agent confidently wrong. Refresh on a schedule that matches how fast the source changes:
- Detect change cheaply. Compare content hashes, use sitemap
lastmodvalues to prioritize revisits, and send conditional requests where the server supports them. - Replace by URL. Delete or tombstone chunks for pages that now return 404 or 410, so they can't be retrieved.
- Re-embed only what changed. The cost of a re-run should scale with changes, not with corpus size.
- Version the pipeline. Store parser version, chunking config and embedding model per build, so you can tell why two runs differ.
For pages that change without announcing it, WebDoppler provides monitoring with webhook alerts, so a change can trigger a targeted re-ingest rather than a full recrawl. See the WebDoppler page for details.
Controlling the bill
Surprise bills usually come from three places: retries on blocked pages, crawls that wander past their useful scope, and a separate per-token charge for LLM extraction. Counter each one:
- Cap crawl depth and page count before you start. Our guide to polite crawling defaults covers 429 handling and crawl caps.
- Back off on rate limits instead of hammering. See backoff and spending caps for rate limits.
- Track cost per ingested record, not just pages per second. We go through the arithmetic in measuring RAG ingestion cost per record.
When your pipeline needs structured fields rather than prose (prices, filings, product specs), Extract returns schema-guided JSON from a prompt or schema. Structured web data for RAG explains when to embed JSON records versus markdown chunks.
Verify before you trust the index
An index that loaded without errors is not an index that retrieves well. Write a set of real questions with the passages that should answer them, run them against the index, and check whether the right chunk lands in the top results. After every change to chunking, parser or embedding model, run the set again. A playbook for building 50 to 150 golden queries walks through assembling that set.
Our perspective
Most ingestion bugs we hear about are not vector database bugs. They are silent fetch failures: a blocked page that became an empty chunk, a redirect that got embedded as a login wall, a crawl that ran longer than anyone intended.
That is why every Gyrence primitive returns a typed response with the failure cases spelled out, and why spending caps and bundled LLM extraction are part of the design. Your agent should be able to reason about what it got. Your bill should not be a discovery.
If you are wiring web data into LangChain, LlamaIndex or a custom MCP client, start by making failure visible. The embeddings take care of themselves after that.
Glen
Start ingesting with Gyrence
Gyrence gives you Search, Traverse, Fetch, Extract and Map behind one API, plus a hosted MCP endpoint for agent clients. There is a Free plan to start on, a Standard plan at $75 per month, Growth at $299 per month and Scale at $549 per month, along with Pay-As-You-Go metered usage. AI extraction carries no separate charge. Compare the tiers on the pricing page.
If you are still scoping the pipeline, read RAG web ingestion explained first, then try a small crawl and measure cost per record.
FAQ
What is the difference between ingestion and indexing?
Ingestion covers everything that gets content into the system: fetching, cleaning, chunking and embedding. Indexing is the vector database's own structure, such as HNSW, that makes similarity search fast. Most retrieval quality problems originate in ingestion.
How big should chunks be?
Anyscale suggests 300 to 500 tokens with 10 to 20 percent overlap as a typical starting point, split along sections, paragraphs or tables. Tune against your own question set, since the right size depends on how your pages are written.
How do I keep the index fresh?
Store a content hash and fetch time per chunk, revisit pages in order of likely change, and re-embed only chunks whose hash changed. Remove chunks for pages that return 404 or 410. For pages that change unpredictably, change monitoring with webhooks can trigger the update.
Do I need to follow robots.txt when ingesting for RAG?
Read it before your first request and honor it. RFC 9309 defines the format and how crawlers should treat errors. For legal questions about specific sites and terms of use, consult your own counsel.
Sources
- Anyscale: RAG basics
- Microsoft generative-pipelines: example RAG ingestion
- RFC 9309: Robots Exclusion Protocol
- OpenAI: Vector embeddings guide
- pgvector on GitHub

