IP reputation and rate models: why slowing down is not enough

IP reputation is the score a bot-detection system gives an address before it reads the request: the network type, the address's history, and how many sessions it has served. Rate limits on protected sites are rarely a fixed number per minute. They are velocity models that count requests per session, per fingerprint and per token.

How an address is scored

flowchart TB
    accTitle: How an IP address is scored
    accDescr: The network class sets a starting penalty, neutral score or bonus, then history, threat feeds, velocity keys and request sequence adjust it into a final score.
    IP[Source IP] --> C{ASN class}
    C -->|datacenter| P1[prior: penalty]
    C -->|residential ISP| P2[prior: neutral]
    C -->|mobile carrier, CGNAT| P3[prior: bonus]
    P1 & P2 & P3 --> H[History: fingerprints seen,<br/>tokens issued, challenges failed]
    H --> TI[Threat feeds: proxy exits, Tor, VPN ranges]
    TI --> V[Velocity keys: per IP, per session,<br/>per fingerprint, per token]
    V --> Q[Sequence shape and concurrency]
    Q --> S{{score}}
    class C violet
    class P1 bad
    class P2 soft
    class P3 ok
    class S key

Slowing down touches one box near the bottom. The class, history and feed boxes above it were decided before your first request arrived.

Three classes of address

Datacenter. Allocated to hosting providers and clouds. The ASN is public. Any request from these ranges is more likely to be automated than human, and the prior reflects it. Some vendors block them outright for consumer sites; most add a score penalty and serve challenges more readily.

Residential. Allocated to consumer ISPs. The prior is neutral. These addresses are obtained by proxy providers through SDKs embedded in free apps, through paid opt-in programs, or through arrangements with ISPs. The quality differs: an address that has served a thousand scraping sessions in a week has a history, and vendors keep that history.

Mobile. Allocated to carriers. Carrier-grade NAT puts hundreds or thousands of subscribers behind one public IPv4 address. Blocking it blocks all of them, so vendors apply a much higher threshold. This is why mobile proxies cost the most.

History and sharing

Beyond the class, vendors keep per-IP state: how many fingerprints have been seen from it, how many tokens were issued, how many were later flagged, how many challenges failed. An address that produced fifty distinct Chrome fingerprints in an hour is a proxy exit, whatever its ASN says. Providers that resell the same residential pool to many customers produce exactly this pattern, and the pool degrades for everyone.

Threat intelligence feeds add another layer: known proxy exit lists, Tor nodes, VPN ranges and addresses observed in credential stuffing. These are shared across vendors and update hourly.

Velocity, not rate

The naive model of a rate limit is "N requests per minute per IP". Real systems measure velocity across several keys:

• Requests per session token, and the ratio of page views to resource loads. Browsers load images, CSS and scripts; a client that fetches only HTML has a ratio of 1:0.
• Sessions per IP per hour, and distinct fingerprints per IP.
• Pages per minute within a session compared with the human distribution for that site. A product page every two seconds for an hour is not a shopper.
• Sequence shape. Humans arrive at a listing, open a few items, go back, filter, and leave. Scrapers walk every page of every category in order. The transition graph is a feature.
• Concurrency. Several open connections from one session fetching different categories at once has no human equivalent.

Slowing a scraper to one page every five seconds fixes only the third bullet.

Rotation as a signal

Rotating the exit address on every request seems to defeat per-IP velocity, and it does, at the cost of creating a new contradiction: the same fingerprint and the same cookies now arrive from a hundred addresses in a hundred cities within minutes. Vendors key velocity on the fingerprint hash as well as the IP, and they flag a session whose address changes country between requests.

The configuration that looks like a person is sticky: one address per session for the session's lifetime, a session that lasts about as long as a human visit, and a fingerprint that stays constant within the session and changes with the address between sessions.

Demonstration: measure a site's velocity model

Without triggering a block, you can estimate how a site counts. Against a site you have permission to test (your own staging environment, or a public practice site such as books.toscrape.com which has no protection, as a control), run a controlled ramp and log status codes:

import time, requests, statistics
URL = "https://example.test/listing?page={}"   # a site you are allowed to test
s = requests.Session()
s.headers["User-Agent"] = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36"
for delay in (5, 2, 1, 0.5, 0.25):
    codes, latencies = [], []
    for page in range(1, 21):
        t0 = time.time()
        r = s.get(URL.format(page))
        latencies.append(time.time() - t0)
        codes.append(r.status_code)
        time.sleep(delay)
    print(f"delay={delay}s codes={sorted(set(codes))} "
          f"p50={statistics.median(latencies):.2f}s max={max(latencies):.2f}s "
          f"retry_after={r.headers.get('Retry-After')}")

Read the output for three things. The delay at which the first 429 or 403 appears is the per-session velocity threshold. A rising median latency before any error is a soft limit, where the site deprioritizes you first.

A Retry-After header tells you the window length. Run it again from a second IP with the same session cookies: if the limit arrives sooner, the site keys on the session, not the address. Run it with a fresh session per request from one IP: if the limit arrives sooner, it keys on the address.

On the unprotected control site every row shows [200] with flat latency, which confirms the harness itself is not the bottleneck.

Frequently asked questions

Why are datacenter IPs blocked more often?

Datacenter address ranges belong to hosting providers, and traffic from them is far more likely to be automated, so many systems start those requests with a penalty or block them outright on consumer sites. Residential addresses start neutral, and mobile carrier addresses start with a bonus.

Are mobile proxies better than residential proxies?

For bot scores, usually yes. Mobile carriers put many subscribers behind one public address, so blocking it would block real people, and vendors set higher thresholds. Mobile proxies also cost the most, and sourcing practices vary between providers.

Should I rotate proxies on every request?

Usually not. When the same fingerprint and cookies arrive from many addresses within minutes, vendors flag the session. One sticky address per session, with the session lasting about as long as a real visit, looks more like a person.

Does slowing down a scraper avoid rate limits?

Only partly. Slower requests fix pages per minute, but velocity models also count sessions per IP, fingerprints per IP, the ratio of pages to resources, crawl order and concurrency, and none of those change when you add a delay.