Intercept the internal API instead of scraping the HTML

Scraping a site's internal API means calling the JSON endpoints its own frontend uses instead of parsing the rendered HTML. The data arrives typed and paginated, often with fields the page never shows, and you can find the endpoint in the browser's DevTools Network tab in about a minute.

Two ways to the same data

flowchart LR
    accTitle: HTML scraping versus calling the internal API
    accDescr: The browser loads an HTML shell, runs JavaScript and fetches JSON from an internal API. A DOM scraper reads the rendered cards, while an API client calls the same endpoint and gets rows, hidden fields and pagination directly.
    subgraph page["what the browser does"]
        H[GET /listing HTML shell] --> J[JS runs] --> A[GET /api/v2/search?page=2] --> R[JSON] --> DOM[render cards]
    end
    subgraph dom["DOM scraper"]
        DOM --> SEL[selectors, scroll loop,<br/>virtualized rows] --> OUT1[rows]
    end
    subgraph api["API client"]
        A2[GET /api/v2/search?page=2<br/>with the same auth] --> R2[JSON] --> OUT2[rows + hidden fields + has_next]
    end
    class page blue
    class dom amber
    class api teal
    class A,A2 key
    class OUT1 warn
    class OUT2 ok

The DOM path reads the output of the render. The API path reads its input, which is smaller, typed, and already paginated.

Why it works

A listing page on a modern site is a shell. After load, the frontend calls something like /api/v2/search?q=...&page=2, receives JSON, and builds the cards. The JSON has the price as a number, the date as ISO 8601, the identifier the site uses internally, and sometimes the stock level or seller rating that the card hides behind a hover.

Pagination is explicit, with a cursor or a page number in the request. None of the DOM problems exist: no selectors, no virtualized rows, no lazy images.

Finding the endpoint

1. Open DevTools, Network tab, filter Fetch/XHR, and clear the log.
2. Trigger the action that loads the data: scroll, change page, apply a filter.
3. Sort by size or look for a response with a JSON content type. Click it and read the Preview.
4. Right click, Copy, Copy as cURL. Paste into a terminal and run it. It will work, because it carries the cookies and headers the browser sent.
5. Remove headers one at a time and re-run to find the minimal set. Usually it is a handful: accept, cookie or authorization, sometimes an x-csrf-token and x-requested-with.

GraphQL sites show a single /graphql endpoint with the query in the POST body, or a persisted query referenced by hash. The Payload tab shows the operation name and variables, which is the contract you need.

The auth ladder

From easiest to hardest:

• No auth. Public JSON. Rare on consumer sites, common on older ones.
• Session cookie. The same cookie the HTML uses. Copy it, and respect that it belongs to an account.
• Bearer token. A JWT in the Authorization header, obtained by an earlier call. Find the call that issued it (often /auth/token or embedded in the HTML as __NEXT_DATA__ or a window.__INITIAL_STATE__ object). Note its expiry.
• CSRF header. A value from a cookie or a meta tag that must be echoed in a header. Read it from the page.
• Request signature. A header like x-signature computed from the path, body, timestamp and a key inside the frontend bundle. Reproducing it means reading the bundle's JavaScript. This is where the job becomes reverse engineering, and where a native-code or VM-obfuscated signer (the Shopee class of problem) can make it uneconomic.
• Anti-bot token. The datadome or x-kpsdk-* style tokens from the anti-bot client script post. The API is behind the same edge as the HTML, so the challenge still has to be solved by a browser somewhere.
• Signed identity. Since 2025 some edges accept a Web Bot Auth signature (RFC 9421 HTTP message signatures with a published key) as an alternative to all of the above. It does not get you past a policy that excludes you; it gets you a policy decision instead of a fingerprint decision.

Demonstration: a public practice site

quotes.toscrape.com is a sandbox built for scraping practice. Its /scroll page is an infinite scroll fed by a JSON endpoint. It has no auth and no protection, which makes it a clean illustration of the method.

Open https://quotes.toscrape.com/scroll with the Network tab filtered to Fetch/XHR, scroll down, and watch /api/quotes?page=2 appear. The client, with pagination derived from the response itself:

import requests

s = requests.Session()
s.headers.update({"Accept": "application/json", "User-Agent": "Mozilla/5.0"})
page, rows = 1, []
while True:
    data = s.get("https://quotes.toscrape.com/api/quotes", params={"page": page}).json()
    rows += [{"author": q["author"]["name"], "text": q["text"], "tags": q["tags"]} for q in data["quotes"]]
    if not data["has_next"]:
        break
    page += 1
print(len(rows), "quotes across", page, "pages")
print(rows[0])

Expected output:

100 quotes across 10 pages
{'author': 'Albert Einstein', 'text': '“The world as we have created it ...', 'tags': ['change', 'deep-thoughts', 'thinking', 'world']}

Twenty lines, no HTML parsing, and the has_next field gave the stop condition for free. Compare with the DOM approach to the same page, which has to drive a scroll loop and detect when nothing new arrives.

For a second exercise, pick a site you use that has a bearer token. Find the token in the page's inline state object, find its exp claim by decoding the JWT payload (base64, no verification needed to read it), and note how long you have before the client needs to refresh.

Frequently asked questions

How do I find a website's hidden API?

Open DevTools, go to the Network tab, filter by Fetch/XHR and clear the log. Then scroll, change the page or apply a filter, and look for responses with a JSON content type. Copy as cURL gives you a working request to start from.

Is scraping an internal API better than scraping HTML?

Usually it is faster and more reliable. The JSON is smaller, has typed values and explicit pagination, and needs no selectors or scroll loops. The trade-off is that internal APIs change without notice, so keep a daily contract test.

Why does the API return 403 when the page works?

The request is missing something the browser sends: a session cookie, a bearer token, a CSRF header, a request signature or an anti-bot token. The API also sits behind the same edge as the HTML, so TLS fingerprinting applies to it too.

Is it legal to use a site's internal API?

It depends on the site's terms and your jurisdiction. Internal APIs are not public APIs, and terms of service usually say so. Reading public, logged-out data is treated differently from getting around authentication, so check the terms and get advice before building on one.