Visit Website →
← Back to blog

Crawl to Vector DB: A Pipeline That Stays Traceable

November 10, 2026
Crawl to Vector DB: A Pipeline That Stays Traceable

A crawl-to-vector-database pipeline has five stages: discover URLs, fetch and clean pages, chunk the text, embed the chunks, and write vectors with source metadata to an index. Retrieval quality is mostly decided in the first three. Below is how to build each stage, what to store beside every vector, and how to keep re-crawls and the bill under control.

Gyrence
Build a More Traceable Web Pipeline
Gyrence provides Search, Traverse, Fetch, Extract, and Map primitives, with typed responses and spending caps for agent-ready web data.
Explore Gyrence

TL;DR

  • Crawl, parse, chunk, embed, sync. Treat it as ETL, not a one-off script.
  • Convert pages to markdown first, then chunk by heading. Keep the source URL, retrieval time, title and section heading on every chunk.
  • Re-index incrementally with content hashes, so unchanged pages are never re-embedded.
  • Check robots.txt and the site's terms before you crawl. In the US, legal exposure turns on authorization, and courts differ.
  • Use hard spending caps and measure cost per record, not just pages per second.

Table of Contents

What a crawl-to-vector pipeline does

The crawler turns a site into raw documents. A parser turns those into clean text plus metadata. A chunker splits the text. An embedding model turns each chunk into a vector, and the vector database stores it so a RAG system can retrieve the closest matches for a question.

Stages from crawling a site to RAG retrieval

Vendor quickstarts show the shape. One scheduled web-crawler pipeline ingests a docs site, creates OpenAI embeddings and writes the vectors to Elasticsearch. Its defaults include a 3600-second reindex interval, depth 50, 1000 URLs and a 500 ms throttle. Those numbers are a reasonable template for your own limits: a bounded depth, a URL cap and a polite delay.

The design principle comes from an ingestion-pipeline write-up: what the parser emits and what you decide to embed caps downstream retrieval quality. A better vector database will not fix bad input text. For a longer walkthrough, see our RAG web ingestion guide.

Stage 1: Scope the crawl

Decide what is in bounds before you fetch anything.

  1. Seed URLs. One domain, one section, or a list.
  2. Depth and URL cap. Hard limits, never open-ended.
  3. Allow and deny patterns. Skip login pages, search result pages, tag archives and calendar URLs that generate endless variants.
  4. Sitemap first. Mapping a domain's URL graph from its sitemap is cheaper than discovering pages by following links, and it gives you a deduplicated list to work from.
  5. Canonical URLs. Normalize query strings and trailing slashes so one page does not enter the index three times.

A dedupe step before scraping matters. One published n8n workflow parses a sitemap and deduplicates URLs in Supabase before it scrapes, cleans, embeds and stores content with metadata.

Stage 2: Fetch and convert to markdown

Markdown is the practical intermediate format. It keeps headings, lists, tables and links, which are exactly the structures a chunker wants. One 2026 analysis notes that markdown is a convenient intermediate format for heading-aware chunking, not a finished retrieval corpus. You still have to strip navigation and repeated boilerplate, and keep tables and links when they carry evidence.

JavaScript-heavy pages need rendering first. The open-source doc2vec project pairs a headless browser with readability-style main-content extraction, converts to markdown, then chunks by heading and token limit before embedding.

What the fetch step returns matters as much as the text. A fetch response should carry the final URL after redirects, page title, retrieval time, content type and a source status beside the markdown. Springbrand's guide recommends this, and adds that markdown is not a verified fact set: consequential claims should stay traceable to the retrieved source.

That last point is where failure modes live. A 403, a bot-challenge page or an empty JavaScript shell can all come back as "successful" text. If your pipeline embeds a captcha page, your index now answers questions with captcha text. Your fetch layer should return a typed result that separates success from blocked, empty, unsupported content type and timeout, so the pipeline can skip or retry instead of embedding garbage. More on this in our pieces on fetching pages to markdown for LLMs and markdown output in scraping.

Stage 2: Fetch and convert to markdown

Stage 3: Chunk and attach metadata

Split on headings first, then by token limit. Keep a heading breadcrumb with each chunk so "Limits" under "Pricing" is not confused with "Limits" under "API".

Store this beside every vector:

FieldWhy it matters
source_url (final, canonical)Citation and deduplication
retrieved_atFreshness filtering
title, section_headingContext for the reader and the model
content_hashIncremental updates
content_type, fetch_statusDebugging and audit
chunk_indexReassembling neighbors

If your use case needs structured fields (prices, dates, filing numbers) rather than prose, run schema-guided extraction to typed JSON and store those values as filterable metadata. Mark each value as sourced from the page or inferred, so a retrieval answer never presents a guess as a quote. See structured web data for RAG.

Stage 4: Embed and write to the vector store

Any embedding model and any store with vector search will work. A production example chunks content, embeds it with Voyage AI and stores chunks and embeddings in MongoDB Atlas Vector Search. Others use Elasticsearch or Postgres with pgvector.

Two rules matter more than the store you pick:

  • Use one embedding model per index. Changing models means re-embedding everything.
  • Upsert by a stable ID. Derive the ID from the canonical URL plus chunk index so a re-crawl overwrites instead of duplicating.

If you also care about how your own pages fare in vector-based retrieval, how vector search affects content is worth a look.

Stage 5: Refresh without re-paying

A pipeline that re-embeds the whole site on every run is slow and expensive. A staged design (discovery, collection, extraction, validation, chunking, indexing, refresh) lets you use last-modified timestamps, content hashes and checksums to skip unchanged pages.

In practice:

  1. Hash the cleaned markdown, not the raw HTML. Ad rotation and timestamps change HTML constantly.
  2. Compare to the stored hash. Same hash: skip. Different: re-chunk and re-embed only that page.
  3. If a URL returns 404 or 410, delete its vectors (a tombstone), or retrieval will keep citing pages that no longer exist.
  4. Log every non-success fetch status. A rising block rate is a signal, not noise.

For pages that change without announcement, monitoring with webhook alerts beats blind scheduled crawls. That is what WebDoppler is for.

Crawling legally and politely

Start with robots.txt. RFC 9309 specifies the Robots Exclusion Protocol, including how crawlers should handle errors and caching of the file. Honor it, throttle your requests, and read the site's terms.

On US law, there is no single rule. A Kirkland & Ellis analysis from 2017 explains that Computer Fraud and Abuse Act exposure can depend on whether access was unauthorized or explicitly revoked, and that courts differ on what "exceeds authorized access" covers. That analysis is dated, so if your crawl targets a specific site at scale, or sits behind a login, talk to a lawyer before you ship. Our own notes cover the four axes of scraping legality and ethical scraping with rate limits and PII controls.

What drives the cost

Three line items: fetch (including rendering), extraction or LLM calls, and embeddings plus storage. Most surprises come from retries, unbounded depth and per-token extraction fees stacked on top of crawl fees.

Controls that keep it predictable:

  • A URL cap and depth cap per job.
  • A hard spending cap on the workspace, not an alert you read tomorrow.
  • Incremental refresh by hash.
  • A metric of cost per indexed record. Our post on measuring cost per record shows how, and crawling vs. scraping covers when each is cheaper.

Our perspective

Most crawl-to-vector tutorials show the happy path: fetch, chunk, embed, done. Web data is a storm. Pages block you, render empty, redirect, or change overnight, and a pipeline that cannot tell those cases apart quietly poisons its own index.

That is why we built Gyrence around honest responses. Every call returns a typed, discriminated-union result, failures included, so an agent or a pipeline can branch on blocked or empty instead of guessing. Search, Traverse, Fetch, Extract and Map are separate primitives you compose, and the same tools are reachable through a hosted MCP endpoint.

We also think a scraping bill should never be a surprise. Gyrence has spending caps and predictable cost per call, and LLM extraction is bundled, with no separate AI extraction charge. Your bill doesn't need a post-mortem.

Glen

Run it on Gyrence

A vector-DB ingestion job maps onto the primitives directly: Map or Traverse to scope URLs, Fetch to get clean markdown, Extract for typed JSON fields, then your own embedder and store. Plans are published on the pricing page: Standard is $75 per month, Growth is $299 per month and Scale is $549 per month, and there are also Free, Founders and Pay-As-You-Go options.

Gyrence

Start on the free tier, set a spending cap, and crawl one section of one site end to end before you scale it. Details are in the Gyrence site.

FAQ

How much of a site should I crawl into a vector database?

Only what your users will ask about. Start with one section, cap depth and URL count, and check retrieval quality on real questions before widening. A smaller, cleaner index usually retrieves better than a large, noisy one.

Should I chunk HTML or markdown?

Markdown. Its headings give you natural chunk boundaries, and conversion removes most markup noise. You still need to strip navigation and boilerplate, and keep tables and links that carry evidence.

How often should I re-crawl?

Match the page's rate of change. Docs might refresh daily, news hourly, policy pages weekly. Whatever the schedule, compare content hashes so only changed pages are re-embedded.

What happens when a page blocks the crawler?

Your fetch layer should return a distinct blocked status, and the pipeline should skip embedding that page. Retry with backoff and log it. Do not try to bypass access controls: see our note on EDGAR scraping for a concrete case.

Sources