How Many Proxies Do I Need? The Actual Math

Derive the number from request volume, per-IP tolerance and concurrency: worked examples, the retry multiplier, and when more IPs fix nothing.

"How many proxies do I need" is the most common pre-sales question in this business, and the standard answer, "it depends", is a cop-out. It does depend, but on exactly three numbers, and you can estimate all three before spending anything. The right count is not a feeling, and it is not a tier on somebody's pricing page. It is the output of a short calculation you can do on paper.

This guide does the calculation properly: the formula, worked examples at two different scales, the correction factors that inflate the naive answer, what changes completely when the product is a rotating gateway rather than a static pool, and the case where the correct number of additional proxies is zero because addresses were never the constraint.

What decides how many proxies you need?

Three numbers, and nothing else until the corrections later on: how many requests the job really makes, how many one address can make before the target objects, and how long you have to finish.

Request volume over a period. How many requests your job actually makes, counted honestly. This is where most estimates fail before any division happens: people count "pages" when each page costs a listing request, a detail request, and two API calls the page fires behind the scenes. If you paginate, every page of results is a request. If you fetch images or scripts, each is a request. If you log in first, the session dance is several more. An honest volume number is routinely three to five times the first guess, and every downstream number inherits the error.

The per-IP rate your target tolerates. The sustained request rate one address can maintain against your specific target before it starts answering with 429s, 403s or challenge pages. Call it r. This is the input everyone wants a provider to supply, and no provider can: it is a property of the target's defenses, not of the proxy. A tolerant API might accept a request per second from one address indefinitely. A heavily defended retail site might object to one request every ten seconds. The spread between those two assumptions moves your final answer by more than an order of magnitude, which is why measuring r (covered below) is worth an afternoon before you buy anything.

Concurrency. How many requests must be in flight at once for the job to finish inside its deadline. Unlike the other two, this one is derived rather than chosen, and it constrains you through a different mechanism than rate does. We will separate the two carefully, because conflating them is how people buy the wrong thing.

What is the formula for sizing a proxy pool?

For a static pool, the sizing calculation is one division:

addresses needed = peak aggregate request rate ÷ per-address tolerated rate

Both sides expressed in requests per second. If you would rather not do the division by hand, the proxy requirement calculator runs every step in this article and shows its working. The important word is peak. A target experiences your rate during the window you run in, not averaged over the calendar. A job that makes all its requests in a nightly four-hour window presents six times the rate of the same volume spread evenly across the day, and it is the four-hour rate the defenses react to. Size for the window you actually use.

Worked example one: a daily price check

The job: 200,000 product pages, refreshed once per day, inside an eight-hour window, a typical price monitoring shape.

Fourteen is the floor, not the order. Some addresses arrive cold and need a gentle ramp, some will trip a defense and need resting, and the day the catalog grows, your volume grows with it. Twenty is a sensible purchase. As a sanity check, run the numbers backwards: 200,000 requests across 14 addresses is roughly 14,000 each, and 14,000 requests at one every two seconds takes about eight hours: the window, exactly. When a sizing answer does not survive being run backwards, one of the inputs is wrong.

Worked example two: a weekly full crawl

The job: five million pages, refreshed weekly, allowed to run through a 48-hour weekend window.

Notice the sensitivity. Halve the tolerance and the answer doubles past 230. Shrink the window to a single night and it triples. But double the page count (the number people fixate on) and it merely doubles. The volume moves the answer no faster than the two numbers nobody bothers to establish. That asymmetry is the most useful thing in this article: r and the window dominate the result, and both are cheap to measure.

Do I need more proxies, or more concurrency?

They are separate floors and you have to clear both, which is why a pool that satisfies the rate arithmetic can still miss the deadline.

The rate arithmetic can be satisfied and the job still miss its deadline, because concurrency constrains you separately, in two ways.

First, defended targets cap simultaneous connections per address, often at one or two, independent of request rate. If your deadline needs 500 requests in flight and the target allows two per address, you need 250 addresses even when the pure rate division said 40. The stricter constraint wins.

Second, latency couples rate to concurrency: your achievable rate equals requests in flight divided by the average duration of one request, a number worth measuring rather than assuming. If each request takes two seconds end to end, 100 workers can never produce more than 50 requests per second, and no quantity of proxies raises that ceiling; only more workers or faster requests do. When throughput falls short of plan, check whether you are address-starved or worker-starved before buying anything, because the two look identical from a distance and only one of them is fixed with money.

Your answer is the larger of the two floors: take the maximum of (rate ÷ r) and (required in-flight ÷ per-address concurrency limit).

How much do retries add to the number of proxies I need?

They multiply it, by whatever your real first-attempt failure rate turns out to be, so a job failing one request in five needs roughly a quarter more capacity than the clean arithmetic suggests.

Everything above assumed each request succeeds on the first attempt. Production disagrees. Timeouts, connection resets, challenge pages and transient server errors (the error code taxonomy sorts out which is which) all convert one logical page into several physical requests.

The correction is expected attempts per success. If one attempt in five fails, each page costs on average 1 ÷ 0.8 = 1.25 attempts. If one in three fails (not unusual against a hostile target), it is 1.5. Failures also cluster: a struggling target fails more, and a naive retry loop pours its retries into exactly the minutes the target can least absorb them, which raises the failure fraction further. Retry with backoff and cap the attempts; web scraping best practices covers that discipline in detail.

Multiply your volume by the retry factor before doing the division. If you have no failure data yet, 1.3 to 1.5 is a defensible planning range for a defended target; replace it with the measured value after the first real run. Skipping this correction is why pools that were sized correctly on paper run out of headroom in week two.

Static pool or rotating gateway: the question changes shape

Everything so far prices a static pool: you buy a specific count of addresses, they appear on your dashboard, and your scheduler shards work across them. That is the natural fit when r is reasonably generous and the target mostly cares about pace; see datacenter proxies for what the static products look like.

A rotating gateway inverts the model. You connect to one hostname, a pool sits behind it, and the exit address changes on each request, or holds still for a sticky session, a distinction sticky versus rotating sessions unpacks. The crucial commercial difference: you are no longer buying an address count at all. The pool supplies the diversity that N was standing in for. What you are buying is bandwidth, so the sizing question stops being "how many" and becomes "how much":

bandwidth needed = requests × average bytes transferred per request

Two million requests at a measured average of 400 KB per request is roughly 800 GB for the run. The number to be suspicious of is the 400. An HTML document alone might be a tenth of that; the same page fetched with images, fonts and scripts might be several times it, and compression settings swing it further. Measure your real average transfer with your real fetch configuration before converting requests to gigabytes, because the gap between "HTML only" and "everything" is the gap between a sensible budget and a shock. The IP rotation guide covers how the rotation itself behaves once you are on it.

A serviceable rule of thumb for choosing between the models: if your problem is pace against a target that tolerates a known rate, a static pool you shard yourself is usually the economical shape. If your problem is that individual addresses keep getting classified and burned, rotating proxies move that problem into the pool, and you budget in gigabytes instead of address counts.

Will buying more proxies stop the 429s?

Only if the limit is keyed to the source address. Limits keyed to an account, an API key, a session cookie or a fingerprint follow your traffic onto every exit you buy, and the diagnostic that tells you which you have takes a few minutes.

Here is the most common failure pattern in proxy purchasing. A job starts collecting 429s. The operator concludes "not enough proxies," triples the pool, and collects exactly the same 429s.

More addresses help only when the limit is keyed to the address. Many limits are not. Rate limits keyed to an account, an API key, a session cookie or a browser fingerprint follow your traffic across every exit you route it through. Sharding an account-keyed limit across 300 addresses achieves nothing except a larger invoice.

The diagnostic costs a few minutes: run the same request pattern through two different exits simultaneously. If each exit earns its own allowance (combined throughput roughly doubles), the limit is per-address, and more addresses genuinely raise your ceiling. If the combined total stays where it was, the key is elsewhere, and the fix is pacing, session handling or header hygiene rather than procurement. Why is my IP getting blocked walks through working out which signal a target is actually keying on.

The same logic applies to fingerprints. If what the target flags is your TLS handshake or header ordering, a thousand addresses carrying the same fingerprint are a thousand blocked addresses. Fix the fingerprint first; then re-derive N, which will have shrunk.

How do I find out how many requests a target allows per IP?

Almost no target documents its limits, and the few that do enforce something different in practice. So measure:

  1. From a single address, request at a deliberately gentle pace (say one request every ten seconds), long enough to trust that the target is genuinely content.
  2. Step the rate up gradually, holding at each level, until the first 429, 403 or challenge page appears.
  3. Take half of the last clean rate as your working r. The headroom is deliberate: limits are typically enforced over bursts and windows rather than smooth averages, and an estimate sitting exactly at the measured edge trips on every minor fluctuation.

Re-measure occasionally, and measure per target. Defenses get retuned without announcement, an r established in a quiet month may not survive the target's own peak season, and no two targets share a number.

Multiple targets share nothing

If the job spans several targets, resist the urge to average. Each target has its own r, its own concurrency cap and its own keying, so run the calculation per target and sum the results. A pool sized to the average is oversized for the tolerant targets and undersized for the defended one, and undersized exactly where it hurts. Sharing one pool across targets also couples their reputations: addresses that picked up flags on an aggressively probed target can arrive pre-classified at the next one when defenses subscribe to shared blocklists. Where the budgets differ wildly, separating the pools is cleaner than letting the hardest target's failures ration everyone else.

Putting the whole calculation together

  1. Count true request volume, including pagination and the per-page request multiplier.
  2. Choose the run window and convert volume into peak requests per second.
  3. Measure r against the real target; do not guess it.
  4. Compute both floors: rate ÷ r, and required in-flight ÷ per-address concurrency.
  5. Take the larger floor and apply the retry multiplier.
  6. Decide the product shape: a static pool if the output is an address count you will shard yourself, a rotating gateway if you would rather buy bandwidth and let the pool carry the diversity, converting requests to gigabytes with measured transfer sizes.
  7. After the first production run, replace every estimate with its measured value and re-derive. The second sizing is always better than the first.

At the polite end, the answer is usually smaller than people fear: plenty of monitoring jobs that "needed" fifty addresses run comfortably on ten. At the defended end it is larger than people hope, because a stingy r and the retry multiplier compound. Either way, it is now a number you derived, so when the target changes its behavior you re-derive it instead of guessing again. When you have yours, pricing maps it onto the available plan sizes for both models.