Visit Website →
← Back to blog

Infinite Scroll Scraping: Find the Endpoint First

October 28, 2026
Infinite Scroll Scraping: Find the Endpoint First

To scrape an infinite scroll page, don't scroll it. Open the Network tab, find the request the page fires when it needs more items, and replay that request with a changed cursor or page number. Scroll a real browser only when the data is assembled client-side and no endpoint exists. Either way, define "done" before you start, or you will ship duplicates and silent gaps.

Gyrence
Turn Web Pages Into Structured Data
Gyrence’s API can fetch pages, extract structured JSON, and map a domain’s URL graph for agent-ready web data.
Explore the API

TL;DR

  • Infinite scroll is JavaScript fetching more data as you near the bottom. Underneath, it is almost always an HTTP request you can replay.
  • Order of preference: JSON endpoint, then HTML-fragment endpoint, then form POST, then a headless browser.
  • Stop on a hard signal (empty response, hasNext: false, no cursor). If you must infer the end, require several stable rounds, not one.
  • Deduplicate on a stable item key and log every stop reason, so a truncated crawl never looks like a complete one.

Table of Contents

How infinite scroll actually loads data

A usability study of scrolling interfaces defines infinite scrolling as asynchronously fetching data and inserting it into the DOM as the user consumes content, typically with JavaScript and a threshold near the bottom of the page. That description tells you where to look.

Most sites implement the threshold with the Intersection Observer API, which MDN lists as a common way to build infinite-scrolling sites. A sentinel element near the bottom enters the viewport, a callback fires, and a fetch or XHR call goes out. A "load more" button does the same thing on click.

Three loading patterns cover nearly everything you will meet:

PatternWhat the page requestsHow you scrape it
JSON API/api/items?cursor=abc returning JSONReplay the request directly
HTML fragmentAn endpoint returning markup that the page appendsCall it, parse the fragment
Client-side assemblyData embedded or built in JS, no clean endpointDrive a browser

Step 1: find the request behind the scroll

  1. Open DevTools, go to Network, filter to Fetch/XHR, and clear the log.
  2. Scroll once, or click "load more".
  3. Look for the new request whose response contains the items. Its URL arguments (page, offset, cursor, limit) are your pagination controls.
  4. Copy it as cURL, paste it in a terminal, change one parameter, and confirm the response changes.

Chrome has supported copying a request as cURL for years, with request headers included. Recent DevTools releases also resend Fetch and XHR requests and decode query parameters in the Payload tab (Chrome 152), which shortens this loop further.

If you want the inspection to be repeatable, script it. Playwright's network documentation shows how to log every request and response, and how to wait for the response caused by a specific interaction. Attach that wait before you scroll or click, or you can miss the request. If events seem to go missing, the same docs recommend blocking service workers.

Two cursor behaviors to test now, not in production:

  • Opaque cursors expire or are tied to the session. Replay one an hour later and see if it still works.
  • Page numbers can drift. If new items are inserted at the top while you crawl, page=2 shifts and you get repeats. Cursor-based endpoints are safer.

Step 2: replay, don't render

Once you have the endpoint, a plain HTTP loop is cheaper, faster and easier to reason about than a browser. A sketch:

seen, cursor = set(), None
while True:
    r = client.get(URL, params={"cursor": cursor, "limit": 50})
    r.raise_for_status()
    body = r.json()
    new = [i for i in body["items"] if i["id"] not in seen]
    seen.update(i["id"] for i in body["items"])
    store(new)
    cursor = body.get("next_cursor")
    if not cursor or not body["items"]:
        stop_reason = "end_of_feed"
        break

Note what the loop does not do: it never treats a 200 as proof of success. An endpoint that starts returning a login wall or a bot-challenge page with a 200 status will poison your dataset. We wrote about that in why a scraper stores block pages as data. Validate the shape of every response before you store it.

Pace the loop, too. Hammering a feed endpoint is the fastest way to get blocked, and our notes on rate limits, backoff and spending caps cover how to back off sanely.

Step 3: when you need a browser

Some pages build the list client-side from state or from a signed, rotating request. There, a headless browser is the honest answer. Before committing, run our headless vs HTTP checklist: you may only need the browser to get past a login, then switch to HTTP, as in this Playwright-then-HTTP pattern.

If you do scroll, do it deliberately:

  • Dispatch real wheel events. Playwright's mouse.wheel is meant for manually scrolling the page.
  • Wait for growth, not time. Playwright-focused guidance recommends waiting for actual scroll-height growth instead of fixed sleeps, then letting network activity settle.
  • Count items, not pixels. Item count in the DOM is a better progress signal than scrollHeight on pages that virtualize their lists.
  • Parse once at the end, or in batches, with a normal HTML parser. If you run Scrapy with Playwright, the same logic applies: scroll incrementally, wait, then hand the rendered HTML to your selectors.

Virtualized lists are the classic trap. Libraries that recycle DOM nodes keep only a window of rows in the document, so "parse everything at the end" returns the last 30 items. Harvest after every scroll step and dedupe by key.

Before you build any of this, it is also worth checking whether the site wants automated access at all, and whether its pages are reachable by bots. An AI crawlability audit is a quick way to see how a site presents itself to automated visitors.

Knowing when to stop

This is where most scrapers fail quietly.

Prefer a direct signal. A response with an empty items array, a missing next_cursor, or a hasNext: false flag beats any heuristic. GraphQL-style connections expose exactly this: Facebook's Relay pagination docs model infinite scroll with explicit hasNext and loading state, fetching another page only when a next page exists.

When there is no signal and you are scrolling a browser, one round with no growth is inconclusive. A slow response looks identical to the end. The Playwright guidance linked above suggests stopping only after a streak of stable rounds. Pick a streak length, set a hard cap on total rounds, and record which one ended the run.

A checklist for every run:

  1. Record stop_reason: end_of_feed, max_rounds, error, blocked.
  2. Record item count and the first and last item keys.
  3. Compare with the count the page claims ("1,240 results"), if it shows one.
  4. Treat a run that ended on max_rounds or error as partial, never complete.

Infinite scroll vs. pagination crawling

Pagination is the easier case. Numbered pages give you a stable URL per slice, so you can fan out in parallel and retry a single page. The usability research cited above notes that infinite scroll suits large, flat streams of similarly important content, while pagination fits goal-oriented comparison and finding specific items. In practice that means feeds, timelines and catalogs often scroll, and search results or directories often paginate.

Patterns worth knowing:

  • Page-number loop: increment page until an empty result. Simple, but drifts on live data.
  • Offset/limit: same drift risk, with a tunable batch size.
  • Cursor chain: each response hands you the next token. Sequential, but consistent.
  • Sitemap or URL graph first: if the site publishes a sitemap, enumerate item URLs from it and skip the scroll entirely.

That last one is underused. A domain map often gives you the full item list in one step, and you then fetch each URL individually, which also makes retries idempotent. See how idempotent fetches work for why that matters.

What usually goes wrong

  • Duplicates: overlapping pages, or items re-ranked between requests. Fix with a stable key.
  • Silent gaps: a cursor expires mid-run and the server returns an empty list that looks like the end. Fix by checking the stop reason against the expected count.
  • Block pages stored as data: covered above. Validate shape.
  • Cost creep: running a browser for every page when an endpoint would do. Re-check errors and typed failures before retrying blindly.
  • Unbounded loops: a feed that never ends. Always set a max.

Perspective: our view on scroll scraping

We think the industry over-reaches for headless browsers. A browser is a fine last resort, but it hides failure: a half-scrolled page still returns HTML, and a scraper that treats any HTML as success will report a clean run on a truncated crawl. Our position is that every step of a scrape should return a typed result, including the failure cases, so your code (or your agent) can tell "the feed ended" from "we got blocked" from "we gave up at the round limit." Those are different outcomes and they should never share a status.

We also think cost predictability matters as much as correctness. A scroll loop with no cap is a billing incident waiting to happen, whoever is metering it. Set caps at the loop level and at the account level.

— Glen

Where Gyrence fits

Gyrence gives you five composable primitives: Search, Traverse, Fetch, Extract and Map, plus a hosted MCP endpoint. For paginated and endpoint-driven sources, Map can enumerate a domain's URL graph, Fetch cleans each page to markdown, and Extract returns structured JSON from a prompt or schema, with typed responses on failure. Spending caps keep a runaway loop from becoming a surprise invoice. Plans start with a Free tier, and paid workspace tiers run $75 per month (Standard), $299 per month (Growth) and $549 per month (Scale), plus Pay-As-You-Go. See pricing or start from the Gyrence homepage.

Where Gyrence fits — overview diagram

FAQ

Can I scrape infinite scroll without a headless browser?

Often, yes. Most infinite scroll pages call a JSON or HTML-fragment endpoint you can replay with plain HTTP. Find it in the Network tab, copy it as cURL, and vary the cursor or page parameter. You need a browser mainly when the list is built client-side with no usable endpoint.

How do I know when the scroll has reached the end?

Use a direct signal first: an empty items array, a missing next cursor, or a hasNext flag. If you only have scroll height or item count, require several consecutive rounds with no growth and set a hard round cap. Log which condition stopped the run.

Why does my scraper return duplicate items?

Usually because page numbers or offsets shift while new items are inserted, or because a virtualized list re-renders the same rows. Deduplicate on a stable ID or canonical URL, and prefer cursor-based endpoints when they exist.

Is a "load more" button different from infinite scroll?

Technically they are the same mechanism with a different trigger. A click replaces the scroll threshold, and the request underneath is usually identical. Capture it in the Network tab and replay it.

Sources