Scrapy Proxy Rotation: Middleware, Retries and Sessions
Where rotation belongs in Scrapy's middleware chain, which retries through a gateway change the exit, and how to cycle a static proxy list safely.
Scrapy already knows how to talk to a proxy. HttpProxyMiddleware ships enabled, and putting a URL in request.meta["proxy"] routes that single request through it. Nothing else is required, which is exactly why so many Scrapy proxy setups stop there and then behave badly under load. The questions that matter sit one level up: which component decides that value, at what point in the request's life, and what happens to the decision when the request fails and comes back around as a retry.
Answer those three carelessly and you get the classic failure modes. Retries that hammer the same dead exit until the request is abandoned. A paginated crawl that changes IP between page two and page three and never notices the listings reshuffled underneath it. Throttling tuned as though all traffic left from one address, applied to traffic that actually leaves from many exits.
This guide is about doing rotation properly at the framework level: middleware ordering, retry integration, throttle interplay, session persistence, and the case where what you have is a static list of addresses rather than a gateway. It is Scrapy-specific by design. If you are working with requests, httpx or aiohttp, the mechanics are entirely different and covered in Python proxy integration; nothing there is repeated here.
The one-line version, and why it is not enough
If you only need one request to leave through one proxy, this is the whole answer:
yield scrapy.Request(
"https://example.com/",
meta={"proxy": "http://USERNAME:PASSWORD@gw-rotating_datacenter.node4.io:8083"},
)HttpProxyMiddleware reads that key, pulls the credentials out of the URL, and sends them as a Proxy-Authorization header. Setting http_proxy and https_proxy in the environment before scrapy crawl does the same thing for every request in the process, with no code at all.
Both work, and both fall apart at the same moment: the first time a request fails. A retried request is a copy of the original, meta included, so it carries the same proxy value back through the chain. Against a gateway that is mostly harmless: the gateway picks the exit per connection, and a connection error kills the connection, so the retry opens a new one and draws a new exit (a status-code retry over HTTPS is the exception, covered in the retry section). Against a specific address from a list it means retrying on the exit that just failed, and against a pinned session it means the same thing. Making the assignment a decision that runs on every pass, retries included, is what a middleware is for.
Where should proxy assignment go in Scrapy's middleware chain?
Between the two built-ins that matter, at priority 610 or thereabouts. Number it lower and your failure handler is dead code; number it higher and your assignment lands after the built-in that needed it. The rest of this section is why.
Downloader middlewares sit between Scrapy's engine and its downloader, ordered by the integer you assign in DOWNLOADER_MIDDLEWARES. The ordering is not cosmetic. On the way out, process_request hooks run in ascending order: lower numbers first, closest to the engine. On the way back in, process_response and process_exception hooks run in descending order. A middleware's number therefore decides both what it is allowed to influence and what has already been decided by the time it speaks.
Two built-ins define the slots that matter. RetryMiddleware sits at priority 550 and owns rescheduling failed requests. HttpProxyMiddleware sits at 750; it reads request.meta["proxy"], strips any credentials embedded in that URL, and converts them into a Proxy-Authorization header on the outgoing request.
That geometry dictates where your own proxy middleware goes: between the two, at 610 or thereabouts.
Its process_request then runs before HttpProxyMiddleware's, so the meta["proxy"] value it writes is in place when the built-in converts embedded credentials into a header. You keep the credential handling Scrapy already does correctly instead of reimplementing it.
Its process_exception runs before RetryMiddleware's, because exception hooks fire in descending order and 610 beats 550. This is the detail tutorials often get wrong. RetryMiddleware.process_exception returns a brand-new Request object, and a middleware that returns a request stops the chain: nothing numbered below 550 ever hears about the failure. Park your proxy middleware at 350, as many examples do, and the failure handler you carefully wrote is dead code on every connection error. It looks like it works, because retries still happen; they just happen without your input.
Nothing needs to be disabled or reordered. The built-ins stay at their defaults, and yours slots into the gap between them.
A middleware that assigns the proxy per request
The configuration first. Three settings carry the credentials and endpoint, so they can differ per spider or per environment without touching the middleware:
# settings.py
DOWNLOADER_MIDDLEWARES = {
"myproject.middlewares.Node4ProxyMiddleware": 610,
# RetryMiddleware (550) and HttpProxyMiddleware (750) keep their defaults.
}
NODE4_PROXY_USER = "USERNAME"
NODE4_PROXY_PASS = "PASSWORD"
NODE4_GATEWAY = "gw-rotating_datacenter.node4.io:8083"
RETRY_TIMES = 4And the middleware itself:
# middlewares.py
import logging
import uuid
from scrapy.core.downloader.handlers.http11 import TunnelError
from scrapy.exceptions import IgnoreRequest, NotConfigured
from twisted.internet.error import (
ConnectError,
ConnectionDone,
ConnectionLost,
ConnectionRefusedError,
TCPTimedOutError,
TimeoutError as TxTimeoutError,
)
logger = logging.getLogger(__name__)
PROXY_LEVEL_ERRORS = (
ConnectError,
ConnectionDone,
ConnectionLost,
ConnectionRefusedError,
TCPTimedOutError,
TxTimeoutError,
TunnelError,
)
class Node4ProxyMiddleware:
def __init__(self, user, password, gateway):
if not (user and password and gateway):
raise NotConfigured("proxy credentials are not configured")
self.user = user
self.password = password
self.gateway = gateway
@classmethod
def from_crawler(cls, crawler):
s = crawler.settings
return cls(
s.get("NODE4_PROXY_USER"),
s.get("NODE4_PROXY_PASS"),
s.get("NODE4_GATEWAY"),
)
# outbound ------------------------------------------------------------
def process_request(self, request, spider):
if request.meta.get("skip_proxy"):
return None
session = request.meta.get("proxy_session")
user = f"{self.user}-session-{session}" if session else self.user
request.meta["proxy"] = f"http://{user}:{self.password}@{self.gateway}"
if session:
# Give each pinned session its own throttling slot; see below.
request.meta["download_slot"] = f"session-{session}"
return None
# inbound -------------------------------------------------------------
def process_response(self, request, response, spider):
if response.status == 407:
# The PROXY rejected our credentials. A different exit will not
# help; every retry would 407 identically. Fail loudly instead.
logger.error(
"407 from the proxy for %s: fix credentials, not the target",
request.url,
)
raise IgnoreRequest("proxy authentication failed")
return response
def process_exception(self, request, exception, spider):
if not isinstance(exception, PROXY_LEVEL_ERRORS):
return None # not ours; let RetryMiddleware apply its policy
if isinstance(exception, TunnelError) and "407" in str(exception):
# HTTPS targets tunnel through CONNECT, so a credential failure
# surfaces here as a TunnelError, never as a 407 response.
logger.error("CONNECT refused with 407: credential problem")
raise IgnoreRequest("proxy authentication failed")
if "proxy_session" in request.meta:
old = request.meta["proxy_session"]
request.meta["proxy_session"] = uuid.uuid4().hex[:12]
logger.info(
"session %s hit %s; the retry gets a fresh session",
old,
exception.__class__.__name__,
)
# Return None: RetryMiddleware (550) runs next, sees the exception,
# and reschedules the request, now carrying the new session.
return NoneWalk through what this does, and what it deliberately declines to do.
Two decisions above are Scrapy-shaped versions of general ones. Whether a request should carry a session at all, and how long that session should live, is covered in sticky session proxies; the rotation policy that decides when a fresh exit is worth the churn is covered in rotating proxies for engineers. Neither is Scrapy-specific, and getting them wrong upstream of this middleware will make the middleware look broken.
process_request writes meta["proxy"] on every pass, unconditionally. That matters because RetryMiddleware builds retries with request.copy(), and the copy carries the old meta, including the old proxy assignment. A middleware that politely respects an existing meta["proxy"] value will send every retry through exactly the assignment that just failed. Overwriting on every pass means a retried request flows through your decision logic again instead of inheriting a stale conclusion.
Credential handling is delegated, not duplicated. The username and password ride in the URL, and HttpProxyMiddleware at 750 turns them into the header. If your password contains characters like @, : or %, percent-encode it before it goes into the URL. Malformed credentials produce an error that is indistinguishable from a wrong password, and the 407 guide covers that whole family of failure.
The inbound hooks treat proxy-side trouble as a different species from target-side trouble. The next two sections explain both halves.
Per-spider versus per-request assignment
There are three levels at which a proxy can attach to Scrapy traffic, and each has a legitimate use.
Process-wide, through the http_proxy and https_proxy environment variables that HttpProxyMiddleware reads at startup. This is the bluntest instrument: every request from every spider in the process goes through one proxy, including ancillary fetches you may not have thought about. It is fine for a five-minute experiment and wrong for anything with two behaviors in it, mostly because nothing in the codebase records that it is happening: the configuration lives in whatever shell launched the crawl.
Per-spider, through custom_settings. Each spider declares its own credentials and gateway, and the middleware picks them up in from_crawler. This is the right default when spiders map one-to-one onto targets, because the target's difficulty is what determines the proxy product, and the spider is where knowledge about the target lives. A spider crawling a lenient API can declare shared datacenter proxies and keep costs down; the spider for a heavily defended retail site declares the residential gateway instead. Which tier a given target deserves is a judgment call: the workable heuristic is to start cheap and escalate on evidence of blocking, not on fear of it.
Per-request, through meta. The middleware above already honors two per-request keys: skip_proxy for requests that should go direct (an internal healthcheck, a fetch from your own infrastructure) and proxy_session for requests that must share an exit with each other. Per-request control is also where mixed crawls live: a spider that pulls lightweight listing pages through plain rotation but pins a session the moment it steps into a flow where the target expects one visitor to stay one visitor.
Rotating a proxy list or pool instead of a gateway
Everything above assumes the endpoint is a gateway: one hostname, and a different exit behind it on every connection. Sometimes what you actually have is a list: a set of static datacenter addresses from your dashboard, or a pool you inherited from a previous project. The list-cycling middlewares on PyPI were built for this case, and most of them are fine, but the logic is short enough that owning it is usually better than depending on a package that last saw a release years ago.
The same 610 slot and the same hooks apply. The differences are that the assignment picks from a cycle, and that a proxy-level failure has to evict the address that produced it, because unlike a gateway, a dead entry in a list stays dead until you stop choosing it:
# middlewares.py
import itertools
import logging
from scrapy.exceptions import IgnoreRequest, NotConfigured
logger = logging.getLogger(__name__)
class ProxyListMiddleware:
def __init__(self, proxies):
if not proxies:
raise NotConfigured("PROXY_LIST is empty")
self.alive = list(proxies)
self.cycle = itertools.cycle(self.alive)
@classmethod
def from_crawler(cls, crawler):
# settings.py: PROXY_LIST = ["http://USER:PASS@203.0.113.10:8080", ...]
return cls(crawler.settings.getlist("PROXY_LIST"))
def process_request(self, request, spider):
if not self.alive:
# NotConfigured only means something at startup; raised here it
# surfaces as a download error and gets misfiled. Drop the request.
logger.error("every proxy in PROXY_LIST has been evicted")
raise IgnoreRequest("no live proxies left")
request.meta["proxy"] = next(self.cycle)
return None
def process_exception(self, request, exception, spider):
if not isinstance(exception, PROXY_LEVEL_ERRORS):
return None
dead = request.meta.get("proxy")
if dead in self.alive:
self.alive.remove(dead)
self.cycle = itertools.cycle(self.alive)
logger.warning("evicted %s after %s (%d left)",
dead, exception.__class__.__name__, len(self.alive))
return None # RetryMiddleware reschedules; next pass draws a new entryPROXY_LEVEL_ERRORS is the same tuple from the gateway middleware. Two habits keep this correct in production. Evict on connection-level errors only, never on a 403 or a captcha page, because a target-side block says nothing about whether the address can carry traffic, and evicting on it will empty the list against any site that blocks at all. And re-admit evicted addresses on a timer, or on the next crawl, since an exit that timed out at 03:00 is often fine at 03:10.
Whether a list is the right shape at all is a separate question. Static datacenter addresses are the right buy when the target accepts datacenter ranges and you want a fixed set of identities you control; a paginated crawl on one of them is trivially sticky because nothing ever rotates. Against a target that classifies datacenter ranges, no amount of cycling helps: every entry in the list is in the same reputation bucket, and the crawl collects the same block page from each of them in turn. That is the case for a rotating gateway at the cheap end, or the residential gateway when the target checks address type rather than just address. If you are still evaluating whose gateway to point this at, the criteria worth testing are in how to choose a proxy provider.
How do I make a Scrapy retry leave from a different exit?
After a connection error through a rotating gateway, you do not have to do anything: the retry opens a new connection, and a new connection is a new exit. After a 429 or 503 on an HTTPS target you may need to, because Scrapy keeps the tunnel alive and reuses it. Through a sticky session you always do, because the pinned username sends the copy straight back to the exit that just failed, and the fix is to mint a fresh session id in process_exception before RetryMiddleware builds that copy.
Scrapy's retry machinery is good, and the temptation to replace it should be resisted. What needs your input is not whether to retry but what a retry means when a proxy is involved.
By default, RetryMiddleware retries a couple of times on connection errors and on the status codes in RETRY_HTTP_CODES: 500, 502, 503, 504, 522, 524, 408 and 429. The retried request is a copy of the failed one, scheduled back through the full middleware chain. Two consequences follow.
First: through a rotating gateway, a retry after a connection error is already a rotation. The copy carries the same meta["proxy"] URL, but that URL names the gateway, not the exit, and the failed connection is gone, so the retry's new connection gets a newly picked exit, usually a different one. A request that failed because one exit was slow or dead gets recovered with no additional machinery at all. Status-code retries are the exception. The gateway chooses an exit per connection: an HTTPS tunnel keeps one exit for its whole life, and Scrapy pools tunnels and reuses them, so a retried 429 or 503 on an HTTPS target can go back out through the same tunnel and the same exit. (Plain-HTTP requests get a new exit per request either way.) If a status-code retry must change exit, send Connection: close on those requests so each tunnel ends with its response; it costs a fresh CONNECT and TLS handshake per request, so verify the effect with an IP-echo request before relying on it.
Second: sticky sessions break that property, deliberately. A session-pinned username asks the gateway for the same exit every time it appears, so a retried copy carrying the same session identifier goes straight back to the exit that just failed. That is what the process_exception hook is for. On a proxy-level error it swaps proxy_session for a fresh identifier before RetryMiddleware builds the copy. Remember the ordering argument from earlier; this is why the middleware must sit above 550. The request survives with its callback, its cb_kwargs and its place in the crawl intact, and only the session dies. That is the right trade: the request's position in your crawl is expensive state, and the session identifier is twelve cheap characters.
The one knob worth turning is RETRY_TIMES. The default of 2 was tuned for direct connections, where a failure usually means the target is down and a third attempt rarely changes the verdict. Through a rotating pool, each attempt after a connection error is close to independent (a different exit, a different route), so a slightly higher ceiling such as 4 buys real coverage: if four separate exits all fail against a target, the problem is almost never the exits, and what remains is either the target or your configuration, neither of which retrying can fix.
What is the difference between a 407, a connection error and a 403?
A 407 is the proxy rejecting your credentials, a connection error is infrastructure between you and the target misbehaving, and a 403 is the target rejecting your request. Only the middle one is worth retrying unchanged.
A proxy-using crawl has three distinct failure families, and collapsing them into one generic retry policy is how error budgets quietly disappear.
A 407 is the proxy refusing your credentials. It is deterministic: if one request draws a 407, every request will, because authentication happens before an exit is ever chosen. Rotating changes nothing, and retrying converts a configuration mistake into a self-inflicted storm of failures. That is why the middleware raises IgnoreRequest on the spot instead of letting the retry chain have it. Note the HTTPS wrinkle: for https:// targets, the request reaches the proxy as a CONNECT tunnel, and a credential rejection arrives as a TunnelError exception rather than as a response with status 407. A handler that only checks response.status will misfile every credential failure on an HTTPS crawl as a flaky connection. The exception hook above inspects tunnel errors for the embedded 407 before treating them as ordinary connection failures. Actually resolving the credential problem (encoding, wrong product, whitelist conflicts) is its own guide.
Connection-level errors are the transient family. Refused connections, handshake timeouts, transfers dropped midway: some exit, route or middlebox misbehaved, and the correct response is a retry on different infrastructure, which the session-rotation hook provides for pinned traffic and the gateway provides for free on rotating traffic.
A 403 is the target's opinion of your request, and so is a 200 whose body turns out to be an interstitial or a captcha page, which no middleware can spot for you without a content check. A fresh exit sometimes helps, but if the trigger was your request fingerprint (header order, missing cookies, inhuman timing), the new exit inherits the block within a few pages. 403 is not in RETRY_HTTP_CODES by default, and adding it should be a deliberate act, because it makes every hard block cost you RETRY_TIMES extra requests of pool bandwidth before it is finally accepted as a block. Before spending money on that, work through why your IP is getting blocked; a 429, by contrast, is the target asking you to slow down, is retried by default, and is a throttling signal, which brings us to throttling.
Should I leave AutoThrottle on when crawling through a proxy pool?
Only if you know what it is reading. AutoThrottle infers server load from observed latency, and through a rotating pool that latency mostly reports which exit you happened to draw, so a run of slow exits throttles a crawl the target was handling comfortably.
Scrapy's politeness controls were designed around an assumption that a proxy pool breaks: that all your requests reach a domain from one address.
Concurrency is governed per download slot. By default the slot key is the target's domain, so CONCURRENT_REQUESTS_PER_DOMAIN (default 8) and DOWNLOAD_DELAY apply to the domain as a whole. Routing through a gateway does not change the key (the target hostname is still the slot), so a rotating crawl of one site remains capped at eight requests in flight, even though the target sees the load spread across many source addresses.
Whether to raise that cap is a question about the target's per-IP tolerance. Rate limits are commonly enforced per source address; rotation divides your request rate across exits, so each individual exit stays far below the threshold that a single address hammering away would trip. That arithmetic is the real reason rotation raises sustainable crawl rates. It is not a license for arbitrary concurrency: per-IP counting is only the crudest detector a target runs, and the others (behavioral analysis, fingerprint checks, account heuristics) see your crawl as one actor no matter how many exits it wears. Raise CONCURRENT_REQUESTS_PER_DOMAIN in steps and watch your block rate as you do it.
AutoThrottle deserves particular suspicion in this setup. It adjusts each slot's delay from observed latency, on the theory that a slowing server needs gentler treatment. Through a rotating pool that theory misreads the data: observed latency is your route to the exit plus the exit's route to the target, and it varies connection to connection because the exit varies. AutoThrottle reads that variance as the "server" straining and stretches its delays: a run of slow exits can throttle a crawl the target was handling comfortably. The choices are to disable it and set an explicit DOWNLOAD_DELAY you have reasoned about, or to keep it as a deliberately conservative default while understanding that it will underestimate what the target can take.
Sticky sessions invert the problem. Several pinned sessions crawling one domain all share the domain's single slot, so they queue behind one another and a per-slot delay serializes traffic that is actually leaving through distinct exits. The fix is the download_slot override you already saw in the middleware: keying the slot by session gives each pinned session its own delay clock and concurrency allowance, which matches the reality that each one is a separate visitor from the target's point of view. The crawl-wide CONCURRENT_REQUESTS total still caps everything, so this cannot run away from you.
How do I keep one exit IP across a paginated crawl?
Mint a session identifier when the browse begins and carry it on every request in that sequence. The middleware appends it to the proxy username, and the gateway keeps routing that credential through the same exit until you change the id.
Some targets bind state to the requesting address: a search cursor, a currency or experiment bucket chosen on first contact, a sorted listing whose page three only means something relative to the page one the same visitor saw. Rotate mid-pagination on a site like that and nothing errors: you silently collect page three of a different shuffle, and the corruption surfaces weeks later as duplicate and missing records nobody can explain. An error would have been kinder.
The session identifier rides in the proxy username: USERNAME-session-a1b2c3 asks the gateway to keep routing that credential through the same exit. One caution before building on it: session pinning is a first-class feature of the residential gateway, and the same middleware works there unchanged; on the rotating gateways a pin depends on the gateway's session store, and when that store is unreachable the request is refused with a 503 whose body says so, rather than quietly served from a rotating exit. Every rotating response also carries an X-Node4-Sticky header (held, opened, none or unavailable), and none on a request you believed was pinned means no session segment arrived, usually because proxy_session was missing from that request's meta (a misspelled segment key is refused with a 407 instead). Verify before a crawl depends on the pin: two requests carrying one id should print one address from an IP-echo service. The middleware builds the username whenever proxy_session is present in meta; the spider's whole job is to mint an identifier when a logical browse begins and carry it to the end of the chain:
import uuid
import scrapy
class ListingsSpider(scrapy.Spider):
name = "listings"
def start_requests(self):
for category in ["lamps", "rugs", "chairs"]:
yield scrapy.Request(
f"https://example.com/c/{category}",
callback=self.parse_page,
meta={"proxy_session": uuid.uuid4().hex[:12]},
)
def parse_page(self, response):
yield from self.extract_items(response)
next_url = response.css("a.next::attr(href)").get()
if next_url:
yield response.follow(
next_url,
callback=self.parse_page,
# meta does not inherit across follow(); pass it on.
meta={"proxy_session": response.meta["proxy_session"]},
)Two details carry the weight here. The session is minted per category, not per spider: sessions should be exactly as wide as the state they protect, so that when one dies it takes a single pagination chain with it rather than the whole crawl. And the meta pass-through in response.follow is explicit, because Scrapy does not propagate meta to follow-up requests on its own; omit that line and every "next page" request quietly reverts to rotation, which is the original silent-reshuffle bug reintroduced by hand.
When an exit dies mid-chain, the exception hook mints a fresh session and the retry proceeds on a new address. Whether that is acceptable depends on the target: for most, resuming on page four with a new identity merely risks one page of inconsistency; for cursor-bound targets, the safe recovery is an errback that restarts the chain from page one. Decide per target, not globally. How long a pinned session actually holds its exit, and when unpinned rotation is simply the better default, is covered in sticky versus rotating sessions.
Pulling it together
The entire apparatus is one middleware of about sixty lines, three settings, and a spider that treats its session identifier as real state. In exchange: retries that land on fresh exits instead of dead ones, credential failures that surface immediately instead of masquerading as flakiness, throttling that matches how the traffic actually leaves, and pagination that cannot silently change identity halfway through.
The middleware pattern can be proven before any plan is on the table: the free tier provisions 3 free shared proxies with 1 GB/month bandwidth and 10 concurrent threads, and pointing PROXY_LIST at those proxies' host and port exercises the machinery above: the ordering, the retry integration, the error classification. What the free tier cannot demonstrate is the gateway itself: its proxies are static shared datacenter addresses, and the rotating gateway refuses credentials whose plan does not include it, so the rotation and session behavior need a rotating plan. Which product a given target actually calls for is a separate question from how to wire it, and it is answered by testing rather than by argument: proxies for Scrapy walks that decision and links the plans. Details are on the pricing page.