Web Scraping Best Practices for 2026

Rate budgets with jitter, conditional requests, resumable jobs, status-class error handling, and catching the 200s that quietly contain nothing.

Scrapers rarely die loudly. The common failure is rot: a selector that stopped matching in March, a block that escalated so gradually nobody noticed, a pipeline that has been filing empty records for six weeks while every status light stayed green. The difference between a scraper that works in a demo and one that is still producing trustworthy data a year later is not cleverness about evasion; it is ordinary engineering discipline, applied to a system whose other half you do not control.

This article is that discipline, in the order it pays off. Proxies appear once, near the end, because they are one component of a working scraper and not a substitute for any of the rest.

Robots and terms are inputs, not decoration

Start where the site tells you how it wants to be crawled. robots.txt is machine-readable and your fetcher should parse it programmatically, not rely on a developer having glanced at it once: honor the disallow rules for your user agent, honor Crawl-delay where present, and read the Sitemap: lines, which are frequently the most efficient discovery mechanism the site offers, handing you a full URL inventory with last-modified dates and sparing you a discovery crawl entirely. The precedence rules are subtler than they look: longest-match wins, Allow can carve an exception out of a Disallow, and a group written for * does not apply to you once a group names your agent. So it is worth checking a specific URL against a specific agent rather than reading the file by eye. Our robots.txt tester does that against a pasted file, which is the same check your fetcher should be making on every URL before it queues one.

The terms of service deserve an actual read by an actual human before a project starts, because they define what you may do with the data, not just how fast you may fetch it. Where the terms and your project conflict, that is a decision for whoever owns the project's risk: made deliberately, up front, not discovered later. Our ten-minute risk screen for ethical scraping is a structured way to make that call before any code ships.

Beyond the legal posture there is an engineering argument that gets less airtime: sites express their tolerance in these files. A crawler operating inside the published rules sits in the band of traffic that operators ignore. Step outside it and you are volunteering for an arms race in which the other side controls the battlefield and you fund both sides of the escalation.

Rate limiting is a budget, and jitter is part of it

Set the limit per target domain, not globally. A global cap of a hundred requests per second protects nobody if all hundred land on one host. Decide what a single site should absorb from you (for most, single-digit requests per second is already generous) and enforce it in the client with a token bucket or semaphore per domain.

Perfectly regular timing is a signature. A request arriving every two seconds to the millisecond looks like exactly what it is, because no organic traffic has that cadence. Randomize the interval around your budget, and back off exponentially when the site signals pressure:

import random, time

def polite_delay(base_seconds: float) -> None:
    # Sleep base ± 50%, so the interval never forms a clean signal.
    time.sleep(base_seconds * random.uniform(0.5, 1.5))

def backoff_delay(attempt: int, cap: float = 120.0) -> None:
    # Exponential backoff with full jitter, for 429/5xx retries.
    time.sleep(random.uniform(0, min(cap, 2 ** attempt)))

When a response carries Retry-After, treat it as authoritative: the site has told you the exact number it wants, and ignoring it converts a temporary slowdown into a durable block. Cool down the whole domain, not just the one URL: pressure signals are almost always host-wide.

Cache locally, and ask conditionally

The politest request is the one you never send, and the second politest is the one that returns no body. Most fetch jobs re-download enormous amounts of content that has not changed since the last run, which wastes the target's compute and your transfer in equal measure.

Two HTTP mechanisms fix this and both predate everyone reading this article. Store the ETag and Last-Modified headers from each response alongside your cached copy; on the next visit send If-None-Match and If-Modified-Since. A 304 Not Modified comes back with no body at all: you keep your cached copy, the target serves a header, and both sides win. On plans where bandwidth is the metered resource (which is how most proxy plans on pricing are structured), every avoided re-download is budget returned to you.

Go one step further and schedule re-visits adaptively. Track how often each page actually changes and let that drive its revisit interval: a product page that updates hourly earns hourly visits, an archive page that has been byte-identical for a month does not. The crawl shrinks to the size of the change rate, which is usually a small fraction of the corpus.

Jobs should be idempotent and resumable

A crash at hour six must not mean restarting at hour zero. Scrapers run long, and something always interrupts them: a deploy, a network wobble, an out-of-memory kill, a spot instance reclaimed. Design for the interruption instead of hoping.

Persist the work queue. Every URL to fetch is a row with a state (pending, in flight, done, failed) in something durable, so a restarted worker picks up exactly where the last one stopped. Give every extracted item a natural key (the product ID, the listing URL, the article's canonical link) and write with upserts, so processing the same page twice updates one record instead of inserting a duplicate. Once re-processing is harmless, retries stop being scary, and most of your error handling collapses into "put it back in the queue".

The same properties buy you horizontal scale for free: workers that pull from a shared queue and write idempotently can be multiplied without coordination, and shut down without ceremony.

Handle errors by status class, not one by one

A scraper that special-cases every status code becomes unmaintainable; a scraper that retries everything melts. The workable middle is a small policy per class:

The dead-letter queue is the unglamorous piece that pays for itself. Anything that failed repeatedly lands there with its history attached, and a weekly skim of it is how you find the site redesign, the new bot wall, or the URL pattern you should stop crawling, while the healthy majority of the job keeps flowing.

The decoy-content problem: monitor for silent failure

The most expensive failure mode in scraping produces no errors at all. The requests succeed, the pipeline hums, and the payloads are worthless: a search results page with zero results, a consent wall where the article should be, a JavaScript shell whose data never rendered, or, nastiest, a page that looks perfectly normal and contains deliberately wrong numbers, served precisely because the site classified you as a bot. Status-code dashboards show unbroken green through all of it. You keep getting 200s full of nothing.

Defending against this means measuring output, not transport:

Validate before the data leaves the pipeline

Extraction and validation are different steps; run both. Define a schema per record type (required fields, types, permissible ranges) and check every record against it at the pipeline boundary. Reject the malformed, but do not discard it silently: quarantine failing records with the page they came from, because the failures are your earliest and cheapest signal that something upstream shifted.

Count nulls per field per run. A selector that breaks does not usually throw; it returns nothing, politely, forever. A field whose null rate moved from two percent to sixty is a broken extractor announcing itself, and a per-field counter is the cheapest instrument that hears it.

Where proxies fit

Everything above assumed requests reach the target; proxies are the component that keeps that true at volume, by distributing your traffic across more than one network identity. They are also only that component: no pool compensates for a scraper with no rate discipline, no cache and no monitoring; it just burns identities faster.

Three decisions matter. Match the proxy type to the target's defensiveness; the datacenter guide covers when cheap hosting-range exits are the right call and when they are pre-emptively refused. Choose a rotation model that matches the workload: per-request rotation through a rotating gateway for stateless fetching, sticky sessions where logins or multi-page flows need one identity to persist. And keep per-exit health visible in your own metrics, because the transport-error clustering described above is how a dying exit announces itself long before a provider dashboard would tell you. Broader workload patterns and which proxy setups fit them are collected on use cases.

The checklist

Before calling a scraper production-ready:

None of it is glamorous. All of it is the difference between a scraper that worked once and a dataset your organization can actually rely on.