← Back to blog

Vector Database Ingestion: Pipeline for Web Data

November 15, 2026
Vector Database Ingestion: Pipeline for Web Data

Vector database ingestion is the pipeline that turns source content into searchable embeddings: fetch, clean, chunk, embed, store with metadata, then keep it fresh. For web data, most failures happen before the embedding step, at fetch and extraction. If a page is fetched badly or cleaned badly, retrieval can't recover it. The fix is to treat the front of the pipeline as typed, observable and cost-capped. That is what we built Gyrence to do.

Gyrence
gyrence.com
Improve the Web Data Before Embedding
Gyrence fetches and cleans web pages, extracts structured data, and returns typed results with spending caps for ingestion pipelines.
Visit

TL;DR

  • Ingestion is five stages: fetch and clean, chunk, embed, upsert with metadata, refresh. Quality lost early is never recovered later.
  • Chunk by structure (headings, paragraphs, tables), not by raw character count, and attach URL, fetch time and content hash to every chunk.
  • Use deterministic chunk IDs and content hashes so re-runs are idempotent and only changed pages get re-embedded.
  • Make failures first-class: a blocked page, a 404 and an empty extraction should be distinct results, not silent gaps in your index.
  • Cap spend per run. Cost per ingested record matters more than raw throughput.

Table of Contents

What ingestion actually involves

According to Anyscale's RAG documentation, ingestion loads documents (PDF, DOCX, PPTX, HTML, TXT), applies a chunking strategy, embeds the chunks and stores them in a vector database. At query time the user's request is encoded to a vector and matched against those embeddings. The top-ranked chunks and their metadata become context for an answer that can cite its sources.

Microsoft's example RAG ingestion pipeline shows the same shape for web pages: download the page, extract text, split it into chunks small enough for the LLM, generate an embedding per chunk, create a collection, and store text, vectors and metadata such as source ID, title and source URL. Its default store is Postgres with pgvector, with Qdrant and Azure AI Search as alternatives.

Six stages of web page vector ingestion

The vector database is the easy part. The hard part is everything that feeds it.

Stage 1: Fetch and clean the page

Web data is noisy: navigation, cookie banners, footers, scripts and duplicated boilerplate. If you embed that, your nearest-neighbor results match on boilerplate instead of content. Anyscale's guidance is blunt on this point: high-quality preprocessing is critical because retrieval can't recover information that was never extracted properly.

What the fetch stage should do:

  1. Respect access rules. Read robots.txt before the first request. The Robots Exclusion Protocol is standardized as RFC 9309, including how clients should handle errors and caching. Our guide to robots meta tags and scraping covers the page-level signals a crawler should also read.
  2. Return clean markdown. Markdown keeps headings, lists and tables, which are exactly the boundaries your chunker wants.
  3. Surface failures by type. A timeout, a block, a redirect and a 404 each call for a different action. A pipeline that collapses them into "empty string" indexes nothing and tells you nothing.
  4. Preserve structure. Stripping too aggressively loses tables and code blocks. See why we preserve HTML during content normalization.

The Gyrence Fetch primitive fetches a page and cleans it to markdown. Every call returns a typed, discriminated-union response, failure cases included, so your code branches on the result instead of guessing. For discovery, Map returns a domain's URL graph via sitemaps, Traverse crawls outward from a starting URL, and Search finds starting points. Together they cover "what pages exist" before you spend anything embedding them.

Stage 2: Chunk by structure

Anyscale suggests typical chunk sizes of 300 to 500 tokens with 10 to 20 percent overlap, preferring semantic or structural splits (section, paragraph, table) over arbitrary cuts. With markdown input, this is straightforward: split at headings, keep the heading path with the chunk, and split oversized sections at paragraph boundaries. Count tokens with your embedding model's own tokenizer, not character estimates.

A chunk with no context retrieves badly. "Set it to 30" means nothing without the heading above it. Carrying the section path as metadata lets you prepend it at embed time or show it in a citation.

Stage 2: Chunk by structure

Stage 3: Embed in batches

Embedding is network-bound, so batch requests and run them concurrently up to your provider's rate limit. Dimension is a storage lever as well: OpenAI's embeddings guide documents a dimensions parameter on its third-generation models that returns shorter vectors. Whatever model you pick, record the model name and version on every chunk. When you change models, old and new vectors are not comparable, and you will need to rebuild the collection.

Stage 4: Upsert with provenance and stable IDs

Store the vector next to the metadata that makes the answer trustworthy. At minimum:

FieldWhy it matters
urlCitation and per-page deletion
fetched_atFreshness filtering and audits
title, section_pathReadable citations, context at embed time
content_hashSkip unchanged chunks on re-runs
token_countBudget checks
embedding_modelSafe re-indexing when the model changes

Derive each chunk ID deterministically, for example a hash of the URL, chunk index and chunk text. Re-running the job then upserts the same rows instead of duplicating them, and a changed page produces new IDs while the old chunks for that URL can be deleted.

If you run Postgres, pgvector keeps vectors in the same database as your metadata and supports HNSW and IVFFlat indexes, which means one transaction can replace a page's chunks atomically.

For compliance and financial-research work, provenance is the product. Every answer should trace to a URL and a fetch timestamp. Our write-up on audit-ready controls for compliance-friendly scraping lists what to log.

Stage 5: Refresh without re-embedding everything

A stale index makes an agent confidently wrong. Refresh on a schedule that matches how fast the source changes:

  • Detect change cheaply. Compare content hashes, use sitemap lastmod values to prioritize revisits, and send conditional requests where the server supports them.
  • Replace by URL. Delete or tombstone chunks for pages that now return 404 or 410, so they can't be retrieved.
  • Re-embed only what changed. The cost of a re-run should scale with changes, not with corpus size.
  • Version the pipeline. Store parser version, chunking config and embedding model per build, so you can tell why two runs differ.

For pages that change without announcing it, WebDoppler provides monitoring with webhook alerts, so a change can trigger a targeted re-ingest rather than a full recrawl. See the WebDoppler page for details.

Controlling the bill

Surprise bills usually come from three places: retries on blocked pages, crawls that wander past their useful scope, and a separate per-token charge for LLM extraction. Counter each one:

When your pipeline needs structured fields rather than prose (prices, filings, product specs), Extract returns schema-guided JSON from a prompt or schema. Structured web data for RAG explains when to embed JSON records versus markdown chunks.

Verify before you trust the index

An index that loaded without errors is not an index that retrieves well. Write a set of real questions with the passages that should answer them, run them against the index, and check whether the right chunk lands in the top results. After every change to chunking, parser or embedding model, run the set again. A playbook for building 50 to 150 golden queries walks through assembling that set.

Our perspective

Most ingestion bugs we hear about are not vector database bugs. They are silent fetch failures: a blocked page that became an empty chunk, a redirect that got embedded as a login wall, a crawl that ran longer than anyone intended.

That is why every Gyrence primitive returns a typed response with the failure cases spelled out, and why spending caps and bundled LLM extraction are part of the design. Your agent should be able to reason about what it got. Your bill should not be a discovery.

If you are wiring web data into LangChain, LlamaIndex or a custom MCP client, start by making failure visible. The embeddings take care of themselves after that.

Glen

Start ingesting with Gyrence

Gyrence gives you Search, Traverse, Fetch, Extract and Map behind one API, plus a hosted MCP endpoint for agent clients. There is a Free plan to start on, a Standard plan at $75 per month, Growth at $299 per month and Scale at $549 per month, along with Pay-As-You-Go metered usage. AI extraction carries no separate charge. Compare the tiers on the pricing page.

Gyrence

If you are still scoping the pipeline, read RAG web ingestion explained first, then try a small crawl and measure cost per record.

FAQ

What is the difference between ingestion and indexing?

Ingestion covers everything that gets content into the system: fetching, cleaning, chunking and embedding. Indexing is the vector database's own structure, such as HNSW, that makes similarity search fast. Most retrieval quality problems originate in ingestion.

How big should chunks be?

Anyscale suggests 300 to 500 tokens with 10 to 20 percent overlap as a typical starting point, split along sections, paragraphs or tables. Tune against your own question set, since the right size depends on how your pages are written.

How do I keep the index fresh?

Store a content hash and fetch time per chunk, revisit pages in order of likely change, and re-embed only chunks whose hash changed. Remove chunks for pages that return 404 or 410. For pages that change unpredictably, change monitoring with webhooks can trigger the update.

Do I need to follow robots.txt when ingesting for RAG?

Read it before your first request and honor it. RFC 9309 defines the format and how crawlers should treat errors. For legal questions about specific sites and terms of use, consult your own counsel.

Sources