Proxies for Scrapy

Set the proxy in a downloader middleware

Assign the proxy in a downloader middleware, before the retry middleware sees the request. Set the proxy in process_request and Scrapy uses it for that attempt; keeping the decision out of the spider means every request gets it, including the ones Scrapy generates itself. The ordering number matters: place the middleware so that a retried request is re-assigned rather than sent back through the exit that just failed.

The chain, the retry interaction and a working class are in the Scrapy middleware guide.

• Set the proxy in process_request, not in the spider, so generated requests are covered
• Order it so a retry is re-assigned rather than reusing the exit that just failed
• Leave the credential in the proxy URL and Scrapy passes it through for you
• Let one endpoint do the rotation instead of maintaining a list in Python

A fresh exit per tunnel or request, or one exit per session

The gateway picks an exit for each HTTPS tunnel your spider opens and for each plain-HTTP request. Scrapy's downloader keeps connections alive, so HTTPS requests that reuse an open tunnel reuse its exit, and a new tunnel gets a new address. For discovery work (search pages, listings, availability checks) that is the behavior you want. For anything with state, hold one exit on purpose: add a session segment to the username and every request carrying that id leaves from the same address, on rotating datacenter for up to 30 minutes from when the pin is first assigned. Residential sessions are best effort, with a 30-minute ceiling: the exit can drop sooner if its device goes offline, so switch to a new session ID after repeated failures. Drop the segment and you are back to rotation with nothing to clean up.

What breaks when the exit changes mid-crawl is covered in sticky versus rotating sessions.

Concurrency decides the bill

Scrapy's concurrency settings control how much traffic you generate, and auto-throttling adapts them to the latency it sees. Through a rotating endpoint that traffic is spread across many addresses, so per-domain settings that looked conservative against a single IP can become a much heavier load, and on a metered product it lands on the invoice. Start lower than you expect, let auto-throttling find the ceiling, run a bounded slice of the crawl, and size the plan from the gigabytes it moved rather than from an estimate.

To size a pool against a page count and a time window, use the proxy calculator.

What you are buying, by product

Residential exits are consumer connections in a partner network, billed by the gigabyte, and are the product to try when a target refuses datacenter ranges. Rotating datacenter selects an exit for each connection (each request, on plain HTTP) from blocks Node4 owns in the United States, Italy and Spain, also billed by the gigabyte, and is the cheaper answer when a target accepts hosting traffic but rate-limits each address. Dedicated datacenter is priced per address with unlimited bandwidth, which suits a heavy crawl of a tolerant target where the bytes would otherwise dominate the cost.

Where each billing unit wins, with the arithmetic, is in unmetered billing versus per GB.