← Back to blog

Newsletter Archive Scraping: A Pipeline That Keeps Sources

November 19, 2026
Newsletter Archive Scraping: A Pipeline That Keeps Sources

Scraping a newsletter archive is a four-part job: walk the paginated list, fetch each issue, clean it to text with its date and canonical URL, then extract structured records. We built Gyrence for that shape of work: Map or Traverse finds the issue URLs, Fetch returns clean markdown, Extract returns typed JSON, and every call, failures included, comes back as a typed response under a spending cap.

Gyrence
gyrence.com
Build a Traceable Newsletter Pipeline
Use Gyrence to map archive URLs, fetch clean markdown, extract typed JSON, and handle failures under a spending cap.
Visit

TL;DR

  • Archives are a list of issue links plus pagination. Walk the list, then fetch each issue page separately.
  • Store the canonical URL, title, date and body for every issue. Keep the source URL attached to anything derived from it, including FAQ pairs.
  • Stop pagination on an empty page or a repeated first issue URL, and skip unchanged issues with a body hash.
  • Treat failures (blocked, paywalled preview, 429) as data, not as empty strings.
  • Cap spend before you start a backfill of hundreds of issues.

Table of Contents

What a newsletter archive looks like to a scraper

A public archive is a list of past campaigns with links to web-version issue pages, sometimes with RSS and sitemap routes. BuildBase documents an archive of this kind with 20 issues per page and an unauthenticated list route that takes a page parameter. Prime Direction's archive shows 12 issues per page and uses either a Load More button or numbered navigation, which matters: a Load More button means JavaScript rendering, numbered pages usually do not.

Before writing any code, look for three things:

  1. A sitemap or RSS feed. Cheapest way to enumerate issues. The Weekly Thing publishes an RSS feed and a search page indexing full issue text.
  2. Stable issue URLs. The same FAQ says its issue URLs use a sequential issue number, which gives you a natural key and a citable reference.
  3. Pagination style. Query parameter, numbered links or a button that loads more.

If the archive has a sitemap, map the domain first and filter to issue paths. If not, traverse outward from the archive index page.

Step 1: Enumerate issues without looping forever

The failure mode in naive loops is pagination that never ends. Some archives redirect an out-of-range page back to page one, so a for page in range(48) loop quietly stores duplicates. Stop when a page returns no issues or when its first issue URL matches one you have already seen, as one Playwright guide recommends.

With Gyrence, Map returns a domain's URL graph from sitemaps, so for many archives you skip pagination entirely. Where there is no sitemap, Traverse starts at the archive URL and moves outward, and the response tells you what it reached and what it did not.

Step 2: Fetch each issue and clean it

Listing pages usually carry only metadata. A Substack-focused RAG guide on DEV Community notes that archive records can provide title, slug, post date, audience, word count, bylines and tags, while the body needs a separate request. The Oracle AI Developer Hub example follows the same pattern: list, fetch each body, convert HTML to text, store canonical URL, title, subtitle and date.

Then strip the chrome. Subscription forms, buttons, SVGs and image wrappers end up in your chunks if you leave them in, and they poison retrieval later. The same DEV guide recommends removing them before indexing.

Fetch cleans a page to markdown for you. If you want to know what "clean" should mean and how much structure to keep, our notes on clean page markdown and content normalization for RAG cover headings, lists and links. Token cost is a reason to prefer markdown too: see HTML vs markdown for LLMs.

Step 3: Record the fields that make data traceable

For compliance and research work, each issue record should carry:

FieldWhy
canonical_urlNatural key and citation target
title and h1Record both and flag mismatches
archive_dateDate shown in the archive
sent_dateLeave null unless the source establishes it
audienceFree or paid, full text or preview
body_hashDetect edits on later runs
fetched_atWhen you saw it

Two judgment calls matter. First, the Playwright guide advises treating an archive timestamp as an archive date, not a sent date, unless you can confirm it. Second, it advises recording the email subject and page H1 separately. Both keep you from asserting more than the source shows. Our post on canonical URL validation covers why a lost or mismatched canonical loses pages.

Extract takes a prompt or schema and returns typed JSON, so these fields come back as a structure rather than as parsed text. There is no separate AI extraction charge.

Step 4: Handle paywalls, blocks and rate limits honestly

A paid post may return only a preview. The DEV guide says to check the audience flag and keep metadata rather than treat the preview as full text. Do not index a teaser as if it were the issue.

Block pages are the other trap. A scraper that stores a "verify you are human" page as an issue body has corrupted your dataset without any error. We wrote about this in your scraper stores block pages as data. Gyrence returns a discriminated-union response, so a blocked fetch is a named failure case your code branches on.

For politeness and resilience, the tap-substack project describes a 0.5-second request throttle, handling of selected 403, 404 and 429 responses, and incremental replication by post date. Copy the idea: throttle, retry on 429, never treat a 404 as an empty issue. The Oracle example keeps listing metadata when a body request returns non-200, which is a sound default.

For reruns, use the canonical URL as the key and compare body hashes so unchanged issues cost nothing to process.

Step 4: Handle paywalls, blocks and rate limits honestly

Step 5: Turn issues into FAQ pairs with evidence

The goal of extracting FAQ pairs is answers you can defend. The documented FAQ-RAG pipeline extracts structured markdown, generates questions answerable from the document, grounds answers strictly in its text, stores source-location metadata with each FAQ and embeds question and answer separately.

FAQ pipeline preserving document evidence

A schema for this in Extract looks like:

{
  "faqs": [{
    "question": "string",
    "answer": "string, only from the issue text",
    "supporting_quote": "string, verbatim",
    "issue_url": "string"
  }]
}

Require a verbatim supporting quote and check that it appears in the fetched markdown. A pair that fails the check is dropped, not shipped. Chunk long issues first; the Oracle example splits text into chunks of about 1,500 characters.

Perspective

Most archive jobs fail quietly. The loop finishes, the table fills, and three weeks later someone finds that 11 issues are block pages and 30 are duplicates of page one.

We designed Gyrence around that problem. Every primitive returns a typed response that includes the failure cases, so your pipeline can branch on blocked, preview only or empty instead of guessing. Search, Traverse, Fetch, Extract and Map compose, so an archive job is a short script, not a stack of vendors.

Cost works the same way. A backfill is exactly where a surprise bill appears, so spending caps and predictable cost per call are part of the product, not an add-on.

Glen

Run it with a cap

Start on the free plan, run one archive end to end, and read the credit cost before you backfill the rest. Paid tiers are Standard at $75 per month, Growth at $299 per month and Scale at $549 per month, and Pay-As-You-Go is available for metered use.

Gyrence

If you are wiring this into an agent, connect over MCP and let the agent call Fetch and Extract as tools. To be told when new issues appear, WebDoppler monitors pages and sends webhook alerts.

FAQ

Can I scrape a newsletter archive without writing pagination code?

Often yes. If the site has a sitemap or RSS feed, enumerate issue URLs from that and skip pagination. Map returns a domain's URL graph from sitemaps, and you filter it to issue paths.

How do I get the real send date of an issue?

Only record it if the source states it. Archive pages usually show an archive date. Store that as archive_date and leave sent_date null otherwise.

How do I avoid duplicate issues?

Use the canonical URL as the key and a hash of the cleaned body to detect changes. Stop pagination when a page is empty or its first issue repeats.

What about paid or paywalled posts?

Check the audience flag. If you only receive a preview, store metadata and mark the body as partial instead of indexing it as the full issue.

Sources