What people actuallybuy proxies for

Almost every scraping block traces to one of three causes, and they need different answers. Per-address accounting is the easiest: the target counts requests per IP and refuses above a threshold, so spreading the same traffic across many addresses fixes it outright. Range rejection is the second: the target refuses hosting ranges on sight, and no quantity of datacenter addresses helps, because the property being rejected is the range rather than the address. The third is behavioral: the target is scoring your headers, your TLS fingerprint and the timing of your requests, and it will keep scoring them whatever address they arrive from.

Naming the three separately matters because buying proxies solves the first two and buying the wrong kind solves neither. Find out which one you have before you spend anything. Every verified account gets a free allocation with no card, so point your existing crawler at your real target through those addresses and read what comes back. If you get the page, datacenter is enough and the cheapest tier that keeps up is the right one. If you get a block or a challenge, that target refuses hosting ranges and residential is the product; a larger datacenter plan returns the same answer more expensively.

A spread of addresses does not raise your request rate by itself, and it is worth knowing why before you tune anything. Scrapy keys its download slots on the target domain, so `DOWNLOAD_DELAY` and `CONCURRENT_REQUESTS_PER_DOMAIN` cap the whole crawl however many exits carry it: the load the target sees is unchanged, it just arrives from more places. The setting that does change this is `CONCURRENT_REQUESTS_PER_IP`, which moves the limit onto the address and quietly disables the per-domain cap. Where the pool really shows up is the invoice: on a per-gigabyte product the bytes land in billing rather than in a log, so measure what one unit of useful work costs on a small run and multiply, rather than discovering the run rate at the end of the month.

What a retail page shows you depends on where it thinks you are. Currency, tax treatment, delivery promise, stock status and the promotion currently running are commonly set by location, and a storefront that redirects a foreign visitor to its local site can show a different assortment as a result. The headline price itself varies less often than teams assume, but everything wrapped around it varies routinely. A price collected from the wrong country is not a rougher answer, it is a wrong one, and nothing in the response says so.

That makes location fidelity the whole requirement here. A residential exit in the market you are measuring returns the page a local shopper is shown, from a connection that reads as an ordinary household rather than a datacenter. Country targeting covers most work. City targeting earns its keep where the thing you are measuring genuinely varies inside the country: grocery delivery coverage, store-level stock, regional promotions.

The second requirement is repeatability, and it is the one that gets skipped. A price series is only useful if the observation conditions hold still, so pin the location, keep the request cadence steady, and record the exit country alongside every price rather than assuming it. When you ask for a location that has no exits, our gateway refuses the request instead of quietly serving a nearby one. A gap in a series is recoverable; a row labeled with a country it did not come from is not.

Marketplace work differs from price monitoring in shape rather than in kind. The unit is the listing rather than the price, the volume is much higher because catalogs are large, and the thing that usually breaks a run is pagination rather than a block. Result sets on a busy marketplace reshuffle constantly as listings arrive and re-rank, so a crawl can duplicate items, miss items, or both, between page two and page three of one query. Changing the exit address does not itself reshuffle the results, and it is worth being clear about that, because the churn happens from a single fixed address too. What an exit change does break is the session carrying your pagination token, and on a marketplace that varies its catalog by country it can swap the catalog underneath you mid-query. Hold one exit for the length of a paginated query and you remove that second failure, not the first. Nothing in the output announces either, which is what makes them expensive.

That is what sticky sessions are for. Adding a session segment to the username pins one exit for the sequence, and dropping the segment returns you to per-request rotation with no state to clean up. Use rotation for discovery, where each request stands alone, and a session for anything paginated, filtered or logged in.

Bandwidth is the other thing that decides whether the project pays for itself. A product page's HTML is usually a small fraction of its images, fonts and scripts, so a client that fetches only the document, or the data endpoint behind it, cuts consumption sharply on a metered product. Measure one listing end to end, then multiply by the catalog, before committing to a tier.

Ad verification is the clearest case on this page for residential addresses, because the thing being verified is a targeting decision made about the viewer. An ad server that recognizes a hosting range can serve a different creative, serve the default, or serve nothing at all, and every one of those looks like a successful check to a script that only counts HTTP 200. Observing from a household connection in the target market is what makes the observation mean what you think it means.

The same logic drives counterfeit and gray-market monitoring. Listings for infringing goods are routinely geofenced, shown to shoppers in one country and hidden everywhere else, and a foreign IP is precisely the visitor a seller hides from. Sampling from inside each market you sell in is the difference between a monitoring program that produces evidence and one that produces a report.

Rotation matters here, but be precise about what it buys. Frequency caps are keyed on cookies and device identity rather than on the source address, so a fresh exit does not by itself reset one; you need a clean profile for that. What a fresh exit does give you is a location and a network that has not already been scored, which is what keeps a check from being filtered as repeat traffic from a known observer before the creative is ever chosen.

Search results are localized well below country level. Local packs, map results, shopping units and the ordering of ordinary organic results all shift between metros inside one country, so a rank measured from a single location is a fact about that location rather than about the query. A team checking from its own office is, quite precisely, measuring its own building.

Volume is the second constraint, and it is the one that decides the product. Rank tracking is many small repetitive requests to one host, which is exactly the pattern per-address accounting exists to stop. Spreading the same schedule across a pool removes the ceiling without increasing the load any single address presents. Search engines challenge hosting ranges harder than most targets, not less, so expect CAPTCHAs and plan for them rather than assuming a datacenter address will sail through. Datacenter is still the usual starting point on cost, and the rotating tiers are built for this shape of traffic; move to residential for the queries where the challenge rate makes the cheaper address the more expensive one.

Where residential earns its place is local intent: a local pack for a suburb, or a result set you want to be certain reflects a consumer connection. Running bulk position tracking on datacenter and the local checks on residential is normally cheaper than putting everything on one product, and it is easy to arrange because both use the same account.

Social platforms rate-limit primarily on the account, token or app rather than on the address, and they treat hosting ranges as automation readily. That combination is why an address change alone rarely buys much here, and why request pacing matters at least as much as the exit: a pool that hides your volume from a rate limiter will not hide a request cadence no human produces. Spread the schedule, and slow it down before you widen it.

Session handling deserves more care here than anywhere else on this page. Anything carrying a persistent identity should stay on one exit for the life of that identity, because an address that jumps between countries between requests is a pattern these platforms specifically look for. Pin a session, keep it, and treat a forced address change as an event your code handles rather than an implementation detail it ignores.

For purely public collection with no identity attached, per-request rotation is the simpler and cheaper default, because there is no state to keep consistent and a blocked exit costs you one retry.

This is the one use case on the page where the target is yours, and that changes the requirement completely. Nothing is trying to block you, so the exit does not need to look like a household connection, and paying residential rates to test your own site is money spent on a property you do not need. What you need is a stable, known address in a chosen location, which is exactly what a shared or dedicated datacenter proxy is.

The work it covers is broad: confirming that a geo restriction actually restricts, checking that a CDN serves the right edge and the right cached variant, verifying currency, language and consent flows for every market you sell into, and confirming that a partner's whitelist really admits the address you gave them. Static addresses matter for the last one in particular, because an address that rotates cannot be whitelisted by anybody.

Whitelist authentication is worth setting up for CI specifically, and it is available on every product, including the datacenter proxies this use case calls for. Registering the runner's address means no credential in the pipeline configuration and nothing to rotate when a contractor leaves. Our proxy endpoints are IPv4, so whitelist the address the runner uses for IPv4 traffic, which is not always the address it prefers for ordinary web requests on a dual-stack network.

Travel is the clearest example of location-dependent pricing on the web. Airlines and online travel agencies quote in the point-of-sale currency, apply different fare rules by market, and run promotions that are visible in one country and absent in the next; hotel rates behave the same way. An aggregator that collects everything from one country has not built a comparison, it has built one country's view of a global market and labeled it a comparison.

The demands this places on a pool are unusual. Breadth of country coverage matters more than depth in any one place, and consistency inside a single quote sequence matters more than raw throughput. The search, select and price steps of a booking engine share server-side state, so a sequence has to hold one exit or the engine loses the itinerary between steps and returns an error that looks like a block. Pin a session for the sequence, rotate between sequences.

Freshness is the other pressure, and it is the one that sizes the pool. Fares move, so a collection window stretched over hours produces a table whose rows are not comparable with each other. Concurrency across many exits is how you compress that window, which means the right question is how long the window may be, not how many routes there are.