Scrapy already knows how to use a proxy. HttpProxyMiddleware ships enabled, and a proxy URL on a request routes it through that address. What Scrapy does not decide for you is which address each request should leave from, what happens to that decision when the request comes back as a retry, and whether a paginated crawl keeps one address for the whole sequence. Those three answers are the difference between a crawl that finishes and one that quietly collects the same blocked page a thousand times.
The one thing that genuinely differs between targets is whether datacenter ranges are accepted at all, and it is cheap to find out. Every verified account gets a free allocation with no card, so point your existing spider at your real target through those addresses and read the response. If you get the page, datacenter is the right product and the cheapest thing that works: start on the smallest shared tier and scale. If you get a block, a challenge, or an empty shell where the content should be, no larger datacenter plan changes that answer, and residential is what you need. Buying more datacenter at that point is spending money to relearn the same fact.
The same decision, walked from the product end, is in choosing the right proxy type.
Assignment belongs in a downloader middleware, before the retry middleware sees the request. Set the proxy in process_request and Scrapy uses it for that attempt; keep the decision out of the spider and every request gets it, including the ones Scrapy generates itself. The ordering number is the part people get wrong: place it below the retry middleware and a retried request carries the dead exit back to the address it just failed on.
The chain, the retry interaction and a working class are in the Scrapy middleware guide.
Per-request rotation is the right default for discovery: search pages, listings, availability checks, anything where each request stands alone. It is the wrong default the moment state exists. A crawl that changes address between page two and page three of one result set is a crawl whose listings can reshuffle underneath it, and a login that moves address mid-flow is one a stateful site can reasonably treat as suspicious. Adding a session segment to the username pins one exit for that session, and dropping it returns you to per-request rotation with no state to clean up.
What actually breaks when the exit changes is covered in sticky versus rotating sessions.
Scrapy's concurrency settings control how much traffic you generate, and its auto-throttling adapts them to the latency it observes. The trap is tuning them as though all traffic leaves one address. Through a rotating endpoint it does not, so per-domain politeness that looked correct against a single IP becomes a much heavier load spread across many, and against a metered product that lands directly on the invoice. Start lower than you think, let auto-throttling find the ceiling, and size the plan from what the crawl actually moved rather than from what you estimated.
To size a pool against a page count and a time window, use the proxy calculator.
Residential exits are real consumer connections, billed by the gigabyte, and are what a target refusing datacenter ranges will accept. Rotating datacenter gives you a fresh address per request from blocks we own, also billed by the gigabyte, and is the cheaper answer when a target tolerates datacenter but rate-limits per address. Dedicated datacenter is priced per address with nothing metering the traffic, which suits a heavy crawl against a tolerant target where the bytes would otherwise dominate the cost.
Why rotation is metered and exclusivity is not is explained in unlimited bandwidth versus per GB.
High-quality residential IPs for accurate, location-specific data
| Bandwidth | Price per GB | Total | Discount |
|---|---|---|---|
| 3 GB | $2.49/GB | $7.47 /mo | - |
| 5 GB | $2.29/GB | $11.45 /mo | Save 8% |
| 10 GB | $1.99/GB | $19.90 /mo | Save 20% |
| 25 GB | $1.69/GB | $42.25 /mo | Save 32% |
| 50 GB | $1.39/GB | $69.50 /mo | Save 44% |
| 100 GBPopular | $1.09/GB | $109.00 /mo | Save 56% |
| 250 GB | $0.89/GB | $222.50 /mo | Save 64% |
| 500 GB | $0.75/GB | $375.00 /mo | Save 70% |
| 1 TB | $0.65/GB | $650.00 /mo | Save 74% |
| 5 TB | $0.49/GB | $2,450.00 /mo | Save 80% |
Auto-rotating shared proxies (up to 5 users per IP), billed by bandwidth usage. Per-request rotation by default, or pin one exit with a session id
| Bandwidth | Price per GB | Total | Discount |
|---|---|---|---|
| 10 GB | $0.59/GB | $5.90 /mo | - |
| 25 GB | $0.52/GB | $13.00 /mo | Save 12% |
| 50 GB | $0.47/GB | $23.50 /mo | Save 20% |
| 100 GBPopular | $0.42/GB | $42.00 /mo | Save 29% |
| 250 GB | $0.37/GB | $92.50 /mo | Save 37% |
| 500 GB | $0.33/GB | $165.00 /mo | Save 44% |
| 1 TB | $0.29/GB | $290.00 /mo | Save 51% |
Test before you decide, because the answer is a property of your target rather than of Scrapy. Run your existing spider through the free datacenter allocation. If the pages come back, datacenter is the cheapest thing that works and the smallest shared tier is the place to start. If they come back blocked or challenged, that target refuses datacenter ranges and residential is the product; a bigger datacenter plan returns the same result.
A rotating endpoint is one hostname that hands you a different exit per request, so there is no list to hold in Python, no health checking, and no code change when the pool changes. A fixed list is worth it when you want to control which address handles which task, for example pinning one address to one logged-in account for its whole life. Scrapy does either; the endpoint is less code.
Add a session segment to the username and reuse it for every request in that sequence. Requests carrying the same session id exit from the same address, and dropping the segment returns you to per-request rotation. On the rotating datacenter tiers a pin is held for up to 30 minutes measured from when it is first assigned, so work that runs longer should mint a fresh session and carry on rather than expecting the original to persist.
They remove one specific cause and leave the others standing. Rotation defeats per-address accounting: rate limits, per-IP request caps and single-address bans stop mattering when requests arrive from different addresses. It does nothing about identity above the IP layer, so cookies, headers, TLS fingerprints and request timing still travel with your client, and a target correlating on those will connect your requests whatever the exit is. Set a real user agent, respect the site, and leave auto-throttling on.
Multiply the pages you intend to fetch by the average response size, then add the assets if your spider follows them, which is where estimates usually go wrong. A crawl fetching HTML only is far cheaper than one pulling images and fonts. On a metered product that arithmetic is the bill, so measure a small run before committing to a large one.
Setup, targeting and troubleshooting, written for people already running this in production.