Scrapy Proxy Middleware: Rotation, Retries, Sessions

Where proxy assignment belongs in Scrapy's middleware chain, how retries land on a fresh exit, and how a session survives a paginated crawl.

Scrapy already knows how to talk to a proxy. HttpProxyMiddleware ships enabled, and putting a URL in request.meta["proxy"] routes that single request through it. Nothing else is required, which is exactly why so many Scrapy proxy setups stop there and then behave badly under load. The questions that matter sit one level up: which component decides that value, at what point in the request's life, and what happens to the decision when the request fails and comes back around as a retry.

Answer those three carelessly and you get the classic failure modes. Retries that hammer the same dead exit until the request is abandoned. A paginated crawl that changes IP between page two and page three and never notices the listings reshuffled underneath it. Throttling tuned as though all traffic left from one address, applied to traffic that actually leaves from a different exit on every request.

This guide is about doing it properly at the framework level: middleware ordering, retry integration, throttle interplay, and session persistence. It is Scrapy-specific by design. If you are working with requests, httpx or aiohttp, the mechanics are entirely different and covered in Python proxy integration; nothing there is repeated here.

Where should proxy assignment go in Scrapy's middleware chain?

Between the two built-ins that matter, at priority 610 or thereabouts. Number it lower and your failure handler is dead code; number it higher and your assignment lands after the built-in that needed it. The rest of this section is why.

Downloader middlewares sit between Scrapy's engine and its downloader, ordered by the integer you assign in DOWNLOADER_MIDDLEWARES. The ordering is not cosmetic. On the way out, process_request hooks run in ascending order: lower numbers first, closest to the engine. On the way back in, process_response and process_exception hooks run in descending order. A middleware's number therefore decides both what it is allowed to influence and what has already been decided by the time it speaks.

Two built-ins define the slots that matter. RetryMiddleware sits at priority 550 and owns rescheduling failed requests. HttpProxyMiddleware sits at 750; it reads request.meta["proxy"], strips any credentials embedded in that URL, and converts them into a Proxy-Authorization header on the outgoing request.

That geometry dictates where your own proxy middleware goes: between the two, at 610 or thereabouts.

Its process_request then runs before HttpProxyMiddleware's, so the meta["proxy"] value it writes is in place when the built-in converts embedded credentials into a header. You keep the credential handling Scrapy already does correctly instead of reimplementing it.

Its process_exception runs before RetryMiddleware's, because exception hooks fire in descending order and 610 beats 550. This is the detail almost every tutorial gets wrong. RetryMiddleware.process_exception returns a brand-new Request object, and a middleware that returns a request stops the chain: nothing numbered below 550 ever hears about the failure. Park your proxy middleware at 350, as many examples do, and the failure handler you carefully wrote is dead code on every connection error. It looks like it works, because retries still happen; they just happen without your input.

Nothing needs to be disabled or reordered. The built-ins stay at their defaults, and yours slots into the gap between them.

A middleware that assigns the proxy per request

The configuration first. Three settings carry the credentials and endpoint, so they can differ per spider or per environment without touching the middleware:

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.Node4ProxyMiddleware": 610,
    # RetryMiddleware (550) and HttpProxyMiddleware (750) keep their defaults.
}

NODE4_PROXY_USER = "USERNAME"
NODE4_PROXY_PASS = "PASSWORD"
NODE4_GATEWAY = "gw-rotating_shared.node4.io:8080"

RETRY_TIMES = 4

And the middleware itself:

# middlewares.py
import logging
import uuid

from scrapy.core.downloader.handlers.http11 import TunnelError
from scrapy.exceptions import IgnoreRequest, NotConfigured
from twisted.internet.error import (
    ConnectError,
    ConnectionDone,
    ConnectionLost,
    ConnectionRefusedError,
    TCPTimedOutError,
    TimeoutError as TxTimeoutError,
)

logger = logging.getLogger(__name__)

PROXY_LEVEL_ERRORS = (
    ConnectError,
    ConnectionDone,
    ConnectionLost,
    ConnectionRefusedError,
    TCPTimedOutError,
    TxTimeoutError,
    TunnelError,
)


class Node4ProxyMiddleware:
    def __init__(self, user, password, gateway):
        if not (user and password and gateway):
            raise NotConfigured("proxy credentials are not configured")
        self.user = user
        self.password = password
        self.gateway = gateway

    @classmethod
    def from_crawler(cls, crawler):
        s = crawler.settings
        return cls(
            s.get("NODE4_PROXY_USER"),
            s.get("NODE4_PROXY_PASS"),
            s.get("NODE4_GATEWAY"),
        )

    # ── outbound ─────────────────────────────────────────────────────────

    def process_request(self, request, spider):
        if request.meta.get("skip_proxy"):
            return None
        session = request.meta.get("proxy_session")
        user = f"{self.user}-session-{session}" if session else self.user
        request.meta["proxy"] = f"http://{user}:{self.password}@{self.gateway}"
        if session:
            # Give each pinned session its own throttling slot; see below.
            request.meta["download_slot"] = f"session-{session}"
        return None

    # ── inbound ──────────────────────────────────────────────────────────

    def process_response(self, request, response, spider):
        if response.status == 407:
            # The PROXY rejected our credentials. A different exit will not
            # help — every retry would 407 identically. Fail loudly instead.
            logger.error(
                "407 from the proxy for %s — fix credentials, not the target",
                request.url,
            )
            raise IgnoreRequest("proxy authentication failed")
        return response

    def process_exception(self, request, exception, spider):
        if not isinstance(exception, PROXY_LEVEL_ERRORS):
            return None  # not ours; let RetryMiddleware apply its policy

        if isinstance(exception, TunnelError) and "407" in str(exception):
            # HTTPS targets tunnel through CONNECT, so a credential failure
            # surfaces here as a TunnelError, never as a 407 response.
            logger.error("CONNECT refused with 407 — credential problem")
            raise IgnoreRequest("proxy authentication failed")

        if "proxy_session" in request.meta:
            old = request.meta["proxy_session"]
            request.meta["proxy_session"] = uuid.uuid4().hex[:12]
            logger.info(
                "session %s hit %s; the retry gets a fresh session",
                old,
                exception.__class__.__name__,
            )
        # Return None: RetryMiddleware (550) runs next, sees the exception,
        # and reschedules the request — now carrying the new session.
        return None

Walk through what this does, and what it deliberately declines to do.

process_request writes meta["proxy"] on every pass, unconditionally. That matters because RetryMiddleware builds retries with request.copy(), and the copy carries the old meta, including the old proxy assignment. A middleware that politely respects an existing meta["proxy"] value will send every retry through exactly the assignment that just failed. Overwriting on every pass means a retried request flows through your decision logic again instead of inheriting a stale conclusion.

Credential handling is delegated, not duplicated. The username and password ride in the URL, and HttpProxyMiddleware at 750 turns them into the header. If your password contains characters like @, : or %, percent-encode it before it goes into the URL. Malformed credentials produce an error that is indistinguishable from a wrong password, and the 407 guide covers that whole family of failure.

The inbound hooks treat proxy-side trouble as a different species from target-side trouble. The next two sections explain both halves.

Per-spider versus per-request assignment

There are three levels at which a proxy can attach to Scrapy traffic, and each has a legitimate use.

Process-wide, through the http_proxy and https_proxy environment variables that HttpProxyMiddleware reads at startup. This is the bluntest instrument: every request from every spider in the process goes through one proxy, including ancillary fetches you may not have thought about. It is fine for a five-minute experiment and wrong for anything with two behaviors in it, mostly because nothing in the codebase records that it is happening: the configuration lives in whatever shell launched the crawl.

Per-spider, through custom_settings. Each spider declares its own credentials and gateway, and the middleware picks them up in from_crawler. This is the right default when spiders map one-to-one onto targets, because the target's difficulty is what determines the proxy product, and the spider is where knowledge about the target lives. A spider crawling a lenient API can declare shared datacenter proxies and keep costs down; the spider for a heavily defended retail site declares the residential gateway instead. Which tier a given target deserves is a judgment call: the workable heuristic is to start cheap and escalate on evidence of blocking, not on fear of it.

Per-request, through meta. The middleware above already honors two per-request keys: skip_proxy for requests that should go direct (an internal healthcheck, a fetch from your own infrastructure) and proxy_session for requests that must share an exit with each other. Per-request control is also where mixed crawls live: a spider that pulls lightweight listing pages through per-request rotation but pins a session the moment it steps into a flow where the target expects one visitor to stay one visitor.

What you should generally not do in 2026 is manage a raw list of proxy IPs inside your own middleware when a gateway product is available. The list-cycling middlewares you find on PyPI were built for static pools: fetch a list of addresses, iterate through them, track per-address health, evict the dead. That is real work, it is state your crawl now has to carry, and against a rotating gateway it is redundant: the endpoint is one fixed hostname, every connection through it leaves from a different exit, and pool health becomes the provider's problem instead of a dictionary in your process. If you are still evaluating whose gateway to point this at, the criteria worth testing are in how to choose a proxy provider. Static datacenter proxies remain the case where a list is genuinely what you have; the same middleware pattern works, with the gateway string swapped for an itertools cycle over the addresses shown on your dashboard.

How do I make a Scrapy retry leave from a different exit?

Through a per-request rotating gateway, you do not have to do anything: a retry is already a rotation. Through a sticky session you do, because the pinned username sends the copy straight back to the exit that just failed, and the fix is to mint a fresh session id in process_exception before RetryMiddleware builds that copy.

Scrapy's retry machinery is good, and the temptation to replace it should be resisted. What needs your input is not whether to retry but what a retry means when a proxy is involved.

By default, RetryMiddleware retries a couple of times on connection errors and on the status codes in RETRY_HTTP_CODES: 500, 502, 503, 504, 522, 524, 408 and 429. The retried request is a copy of the failed one, scheduled back through the full middleware chain. Two consequences follow.

First: through a per-request rotating gateway, a plain retry is already a rotation. The copy carries the same meta["proxy"] URL, but that URL names the gateway, not the exit; the next connection through it leaves from a different address. A request that failed because one exit was slow or dead gets recovered with no additional machinery at all. This is the strongest practical argument for doing rotation at the gateway rather than in your own code: the retry path, the trickiest path in any crawler, becomes correct by construction.

Second: sticky sessions break that property, deliberately. A session-pinned username asks the gateway for the same exit every time it appears, so a retried copy carrying the same session identifier goes straight back to the exit that just failed. That is what the process_exception hook is for. On a proxy-level error it swaps proxy_session for a fresh identifier before RetryMiddleware builds the copy. Remember the ordering argument from earlier; this is why the middleware must sit above 550. The request survives with its callback, its cb_kwargs and its place in the crawl intact, and only the session dies. That is the right trade: the request's position in your crawl is expensive state, and the session identifier is twelve cheap characters.

The one knob worth turning is RETRY_TIMES. The default of 2 was tuned for direct connections, where a failure usually means the target is down and a third attempt rarely changes the verdict. Through a rotating pool, each attempt is close to independent (a different exit, a different route), so a slightly higher ceiling such as 4 buys real coverage: if four separate exits all fail against a target, the problem is almost never the exits, and what remains is either the target or your configuration, neither of which retrying can fix.

What is the difference between a 407, a connection error and a 403?

A 407 is the proxy rejecting your credentials, a connection error is infrastructure between you and the target misbehaving, and a 403 is the target rejecting your request. Only the middle one is worth retrying unchanged.

A proxy-using crawl has three distinct failure families, and collapsing them into one generic retry policy is how error budgets quietly disappear.

A 407 is the proxy refusing your credentials. It is deterministic: if one request draws a 407, every request will, because authentication happens before an exit is ever chosen. Rotating changes nothing, and retrying converts a configuration mistake into a self-inflicted storm of failures. That is why the middleware raises IgnoreRequest on the spot instead of letting the retry chain have it. Note the HTTPS wrinkle, because it catches nearly everyone: for https:// targets, the request reaches the proxy as a CONNECT tunnel, and a credential rejection arrives as a TunnelError exception rather than as a response with status 407. A handler that only checks response.status will misfile every credential failure on an HTTPS crawl as a flaky connection. The exception hook above inspects tunnel errors for the embedded 407 before treating them as ordinary connection failures. Actually resolving the credential problem (encoding, wrong product, whitelist conflicts) is its own guide.

Connection-level errors are the transient family. Refused connections, handshake timeouts, transfers dropped midway: some exit, route or middlebox misbehaved, and the correct response is a retry on different infrastructure, which the session-rotation hook provides for pinned traffic and the gateway provides for free on rotating traffic.

A 403 is the target's opinion of your request, and so is a 200 whose body turns out to be an interstitial or a captcha page, which no middleware can spot for you without a content check. A fresh exit sometimes helps, but if the trigger was your request fingerprint (header order, missing cookies, inhuman timing), the new exit inherits the block within a few pages. 403 is not in RETRY_HTTP_CODES by default, and adding it should be a deliberate act, because it makes every hard block cost you RETRY_TIMES extra requests of pool bandwidth before it is finally accepted as a block. Before spending money on that, work through why your IP is getting blocked; a 429, by contrast, is the target asking you to slow down, is retried by default, and is a throttling signal, which brings us to throttling.

Should I leave AutoThrottle on when crawling through a proxy pool?

Only if you know what it is reading. AutoThrottle infers server load from observed latency, and through a rotating pool that latency mostly reports which exit you happened to draw, so a run of slow exits throttles a crawl the target was handling comfortably.

Scrapy's politeness controls were designed around an assumption that a proxy pool breaks: that all your requests reach a domain from one address.

Concurrency is governed per download slot. By default the slot key is the target's domain, so CONCURRENT_REQUESTS_PER_DOMAIN (default 8) and DOWNLOAD_DELAY apply to the domain as a whole. Routing through a gateway does not change the key (the target hostname is still the slot), so a rotating crawl of one site remains capped at eight requests in flight, even though the target sees the load spread across many source addresses.

Whether to raise that cap is a question about the target's per-IP tolerance. Rate limits are commonly enforced per source address; per-request rotation divides your request rate across exits, so each individual exit stays far below the threshold that a single address hammering away would trip. That arithmetic is the honest reason rotation raises sustainable crawl rates. It is not a license for arbitrary concurrency: per-IP counting is only the crudest detector a target runs, and the others (behavioral analysis, fingerprint checks, account heuristics) see your crawl as one actor no matter how many exits it wears. Raise CONCURRENT_REQUESTS_PER_DOMAIN in steps and watch your block rate as you do it.

AutoThrottle deserves particular suspicion in this setup. It adjusts each slot's delay from observed latency, on the theory that a slowing server needs gentler treatment. Through a rotating pool that theory misreads the data: observed latency is your route to the exit plus the exit's route to the target, and it varies request-to-request because the exit varies. AutoThrottle reads that variance as the "server" straining and stretches its delays: a run of slow exits can throttle a crawl the target was handling comfortably. The choices are to disable it and set an explicit DOWNLOAD_DELAY you have reasoned about, or to keep it as a deliberately conservative default while understanding that it will underestimate what the target can take.

Sticky sessions invert the problem. Several pinned sessions crawling one domain all share the domain's single slot, so they queue behind one another and a per-slot delay serialises traffic that is actually leaving through distinct exits. The fix is the download_slot override you already saw in the middleware: keying the slot by session gives each pinned session its own delay clock and concurrency allowance, which matches the reality that each one is a separate visitor from the target's point of view. The crawl-wide CONCURRENT_REQUESTS total still caps everything, so this cannot run away from you.

How do I keep one exit IP across a paginated crawl?

Mint a session identifier when the browse begins and carry it on every request in that sequence. The middleware appends it to the proxy username, and the gateway keeps routing that credential through the same exit until you change the id.

Some targets bind state to the requesting address: a search cursor, a currency or experiment bucket chosen on first contact, a sorted listing whose page three only means something relative to the page one the same visitor saw. Rotate mid-pagination on a site like that and nothing errors: you silently collect page three of a different shuffle, and the corruption surfaces weeks later as duplicate and missing records nobody can explain. An error would have been kinder.

The session identifier rides in the proxy username: USERNAME-session-a1b2c3 asks the gateway to keep routing that credential through the same exit. One caution before building on it: session pinning is a first-class feature of the residential gateway, and the same middleware works there unchanged; on the rotating gateway the segment is accepted but honoring it depends on a session store being enabled gateway-side, so verify it directly before a crawl depends on the pin: two requests carrying one id should print one address from an IP-echo service. The middleware builds the username whenever proxy_session is present in meta; the spider's whole job is to mint an identifier when a logical browse begins and carry it to the end of the chain:

import uuid

import scrapy


class ListingsSpider(scrapy.Spider):
    name = "listings"

    def start_requests(self):
        for category in ["lamps", "rugs", "chairs"]:
            yield scrapy.Request(
                f"https://example.com/c/{category}",
                callback=self.parse_page,
                meta={"proxy_session": uuid.uuid4().hex[:12]},
            )

    def parse_page(self, response):
        yield from self.extract_items(response)

        next_url = response.css("a.next::attr(href)").get()
        if next_url:
            yield response.follow(
                next_url,
                callback=self.parse_page,
                # meta does not inherit across follow(); pass it on.
                meta={"proxy_session": response.meta["proxy_session"]},
            )

Two details carry the weight here. The session is minted per category, not per spider: sessions should be exactly as wide as the state they protect, so that when one dies it takes a single pagination chain with it rather than the whole crawl. And the meta pass-through in response.follow is explicit, because Scrapy does not propagate meta to follow-up requests on its own; omit that line and every "next page" request quietly reverts to per-request rotation, which is the original silent-reshuffle bug reintroduced by hand.

When an exit dies mid-chain, the exception hook mints a fresh session and the retry proceeds on a new address. Whether that is acceptable depends on the target: for most, resuming on page four with a new identity merely risks one page of inconsistency; for cursor-bound targets, the honest recovery is an errback that restarts the chain from page one. Decide per target, not globally. How long a pinned session actually holds its exit, and when per-request rotation is simply the better default, is covered in sticky versus rotating sessions.

Pulling it together

The entire apparatus is one middleware of about sixty lines, three settings, and a spider that treats its session identifier as real state. In exchange: retries that land on fresh exits instead of dead ones, credential failures that surface immediately instead of masquerading as flakiness, throttling that matches how the traffic actually leaves, and pagination that cannot silently change identity halfway through.

The middleware pattern can be proven before any plan is on the table: the free tier provisions 3 free shared proxies with 1 GB/month bandwidth and 10 concurrent threads, and pointing NODE4_GATEWAY at one of those proxies' host and port exercises the machinery above: the ordering, the retry integration, the error classification. What the free tier cannot demonstrate is the gateway itself: its proxies are static shared datacenter addresses, and the rotating gateway refuses credentials whose plan does not include it, so the rotation and session behavior need a rotating plan. Which product a given target actually calls for is a separate question from how to wire it, and it is answered by testing rather than by argument: proxies for Scrapy walks that decision and links the plans. Details are on the pricing page.