Rotating proxies for web scraping: a setup guide
Set up rotating proxies for a scraper that keeps running at volume: choose the rotation unit, the pool type, retries and a per-target ban-rate rule.
A scraper that sends every request from one address gets throttled once volume picks up, because requests per address over time is the first thing a target counts. Rotating proxies fix that one signal by spreading your requests across many exits. Setting them up well comes down to four decisions: what unit of work gets its own exit, which pool fits the target, how failed requests are retried, and how you notice that a target has started refusing you. This guide takes them in that order.
Rotation does not fix a scraper that looks automated in other ways. A fresh address carrying the same user agent, header order and request rhythm on every call is still easy to spot, so treat rotation as one layer and the request itself as another.
Decide what gets its own exit
"Rotate on every request" sounds simple, but a rotating gateway assigns exits to connections, not to the requests inside them. On Node4's rotating gateways an HTTPS request travels through a tunnel (a CONNECT), and that tunnel holds one exit for its whole life. A SOCKS5 connection works the same way. A plain-HTTP request gets a new exit each time.
That has a practical consequence. HTTP client libraries reuse connections by default, so a Python requests.Session or a Node agent with keep-alive can send hundreds of HTTPS requests through one tunnel, and all of them leave from the same address. If you want a different exit per page, close the connection after each request or give each worker a short-lived connection. If you want the opposite, several requests from one address, keeping the tunnel open is one way and a session ID is the more robust one.
A session ID in the username pins one exit across connections:
curl -x "http://USERNAME-session-cart0142:PASSWORD@gw-rotating_datacenter.node4.io:8083" https://api.ipify.orgOn Rotating Datacenter and Rotating Unmetered plans, that pin holds for up to 30 minutes from first assignment, and activity does not extend it. Residential sessions are best effort, with a 30-minute ceiling, and can end sooner if the device behind the exit goes offline. The window is set by the product, not by you, so size each multi-step job (a login followed by a few pages, a search followed by its result pages) to finish well inside it, and restart the job with a new session ID if the exit changes partway. The sticky versus rotating sessions guide covers detecting that change.
The rule of thumb: independent public pages get a fresh connection each, and anything with state (cookies, a cart, a paginated cursor) gets one session ID per job.
Pick the pool for the target
| Pool | Fits | Watch for |
|---|---|---|
| Rotating Datacenter | Public pages on targets that accept hosting-network traffic, in the US, Italy or Spain | Some targets challenge or filter datacenter ranges outright |
| Rotating Unmetered | The same pool when data volume is large or hard to predict | Priced by concurrent connections, so a slot is held for as long as a tunnel stays open |
| Residential | Targets with strong bot detection, or countries outside those three | Billed per GB, and sessions can end when the device goes offline |
Rotating Datacenter and Rotating Unmetered draw on the same exits, Node4-owned blocks in the United States, Italy and Spain; the difference is the meter. Datacenter is billed by transfer (rotating prices). Unmetered has no data cap and is priced by how many connections you hold open at once (connection-based pricing), which is why the connection behavior above matters for it: an idle keep-alive tunnel occupies a slot just as a busy one does. Network safety limits still apply.
Residential proxies come from a partner network in 170+ countries, targeted with a country segment in the username. They cost more per GB (residential prices), so they earn their place on targets that refuse datacenter traffic, not on every target. Run a sample of real pages through both pools before you commit a target to one; datacenter vs residential proxies walks through the trade-off.
Retry through a different exit
Rotation lowers refusals; it does not remove them. Every scraper needs a failure path:
- Retry a refused request through a new connection, so it gets a different exit, not the one that just failed.
- Cap retries per URL. Three to five attempts is a common limit, and it stops one bad page from looping forever.
- Back off exponentially on 429 and 403 responses instead of retrying immediately.
- Log the status code, the target and the exit address for every failure, so a proxy problem can be told apart from a target that changed its layout or added a challenge.
Keep the request itself plausible while you do this. Use user agents from current browsers, keep accept-language and accept-encoding consistent with the browser you claim to be, and vary request timing rather than firing at fixed intervals.
Track ban rate per target
Ban rate, the share of requests that come back refused or challenged, is the number that tells you whether the setup is working. Track it per target domain. An aggregate figure can look healthy while one domain refuses a large share of your requests.
Pick a threshold and act on it. For example, if refusals on one target climb past 5 to 10 percent, reduce concurrency on that target first. If that does not help, spread the load across more exits (a larger pool or a second country) before you rotate any faster. Rotating faster through a small pool does not help, because the same few addresses keep coming back and the per-address request rate the target sees stays the same. If parallelism rather than refusals is your ceiling, high concurrency proxies covers sizing concurrent connections.
Where to start
Take one target and a few hundred representative URLs. Run them through Rotating Datacenter with a fresh connection per page, log status and exit per request, and compute the ban rate. If it stays under your threshold, scale that target up gradually. If it does not, move that target to residential and compare. Repeat per target rather than choosing one pool for everything. Plans are on the rotating proxies page.