Automatically rotate proxy IPs when a request gets blocked

Why is my IP getting blocked? Diagnose the response first, then rotate only on a confirmed IP block, with rate-limit and auth failures excluded.

Instead of manually changing proxy settings whenever an authorized request fails, use a bounded retry loop that selects the next configured proxy only after a confirmed IP-specific rejection. Keep authentication failures, rate limits and access restrictions out of that rotation path.

TL;DR

• If your IP is being blocked, identify the reason before changing exits.
• Rotate only after a confirmed IP-specific rejection on a target you are authorized to access.
• An HTTP 429 response requires rate-limit handling, not another proxy.
• Changing a proxy endpoint does not prove that your exit IP changed.

Why this matters

Rotation helps when you need controlled exit changes in an authorized data-collection workflow. It changes the network source of a request. It does not repair an expired token, grant access to a protected resource or make a rejected request valid.

A retry loop needs a classification rule, an exit-selection rule and a stopping condition. Without those controls, it repeats the same mistake from different addresses. The rotating proxies for web scraping guide covers the broader workflow; this guide focuses on the failure-handling branch. If you would rather have a gateway handle exit selection than build this loop yourself, see Node4's rotating proxies and its pricing.

Start with one approved target and one worker. Prove the behavior before attaching the loop to a scheduler or a shared queue.

Before you start

• Have authorized target access and proxy credentials. Obtain the endpoint, protocol and authentication details from your provider. Use a test resource whose expected response you know.
• Have Python, pip and an approved block signal. The example uses Requests. The target owner or its documentation must identify a response-body marker that specifically means an IP-based rejection. A generic forbidden page is not enough.
• Check how exit assignment works. Distinct proxy URLs can still reach the same exit, and sticky routing can preserve an address across requests. A new connection alone is not proof of rotation.

This example handles public or otherwise authorized, stateless HTTP GET requests. It deliberately does not carry cookies between attempts. Do not attach it unchanged to a checkout, account-management action or another operation that writes data.

The sample uses 3 total attempts, a 10-second connection timeout, a 30-second read timeout and a 2-second base pause between eligible retries. These are example configuration values, not provider specifications or throughput recommendations.

Proxy configuration

Set the request inputs

1. Install Requests in your project environment:

python -m pip install requests

1. Set the approved destination and proxy list. Enter actual connection strings at the prompts rather than putting credentials into shell history:

read -r -p 'Authorized target URL: ' TARGET_URL
export TARGET_URL

read -r -s -p 'Proxy URLs, separated by commas: ' PROXY_URLS
printf '\n'
export PROXY_URLS

read -r -p 'Documented IP-block response marker: ' IP_BLOCK_MARKER
export IP_BLOCK_MARKER

1. Keep PROXY_URLS as a comma-separated list of complete proxy URLs. This script assumes an HTTP proxy connection string. Authentication and any provider-specific session identifiers belong in the connection strings supplied for your account.

Expected result: your process has a destination, at least one proxy and an explicit rejection marker. Nothing has been sent yet.

Treat proxy credentials as secrets. Environment variables avoid embedding them in source code, but they are not a substitute for your production secret-management policy. Do not print the proxy list in application logs.

Use endpoints with understood exit behavior. If your provider assigns exits through a session parameter, follow its documented syntax. Do not invent a username suffix or assume another provider's convention works.

Check the baseline request

Send one request through the first endpoint before enabling retries:

FIRST_PROXY="${PROXY_URLS%%,*}"
curl --proxy "$FIRST_PROXY" \
  --connect-timeout 10 \
  --max-time 30 \
  --dump-header response-headers.txt \
  --output response-body.txt \
  "$TARGET_URL"

The curl command has a 30-second total limit. That differs from the Python read timeout, which is not a total wall-clock deadline. Keep that distinction in your job scheduler.

Inspect the response headers and body locally. Keep response files out of public repositories because they can contain sensitive target data.

Expected result: you know whether the baseline succeeds, fails at proxy authentication or reaches the destination and receives a rejection. Do not add rotation until that distinction is clear.

Response classification

Define what triggers a retry

1. Accept ordinary successful responses.
2. Stop on proxy authentication failures and target authentication failures.
3. Stop on rate limits and preserve the server's retry instructions.
4. Rotate only when a forbidden response contains your documented IP-block marker.

A status code is evidence of a response category, not proof that the exit IP caused it. HTTP 403 means the server refuses the request. It does not identify the refusal's underlying cause.

A successful status still needs application-level validation. A login page, empty dataset or challenge page can arrive with HTTP 200. Check the expected content before treating the job as complete.

Keep the classifier narrow. Do not match generic words such as denied or blocked. They can appear in unrelated error pages, documentation or ordinary content. Use a specific signal approved for this integration.

Expected result: every response reaches one explicit branch. Unknown failures stop rather than trigger more traffic.

Retry-loop configuration

Add bounded proxy selection

Save this as rotate_on_block.py. The program never uses an endpoint twice within one request operation, although endpoint uniqueness does not guarantee exit-IP uniqueness.

import os
import random
import time

import requests

TARGET = os.environ['TARGET_URL']
PROXIES = list(dict.fromkeys(
    p.strip() for p in os.environ['PROXY_URLS'].split(',')
    if p.strip()
))
MARKER = os.environ['IP_BLOCK_MARKER'].strip()
MAX_ATTEMPTS = 3
CONNECT_TIMEOUT = 10
READ_TIMEOUT = 30
BASE_PAUSE = 2

if not PROXIES or not MARKER:
    raise SystemExit('Proxy list and block marker are required')


def fetch():
    candidates = PROXIES[:MAX_ATTEMPTS]
    for index, proxy in enumerate(candidates):
        with requests.Session() as session:
            session.trust_env = False
            session.proxies.update({
                'http': proxy,
                'https': proxy,
            })
            try:
                response = session.get(
                    TARGET,
                    timeout=(CONNECT_TIMEOUT, READ_TIMEOUT),
                    allow_redirects=False,
                )
            except requests.RequestException:
                raise RuntimeError(
                    'Transport failure; inspect connection settings'
                ) from None

            status = response.status_code
            print(f'attempt={index + 1} status={status}')

            if 200 <= status < 300:
                return response.content

            if status == 429:
                retry_after = response.headers.get('Retry-After')
                response.close()
                raise RuntimeError(
                    f'Rate limited; Retry-After={retry_after!r}'
                )

            if status in (401, 407):
                response.close()
                raise RuntimeError('Authentication failure; stop')

            ip_block = status == 403 and MARKER in response.text
            response.close()
            if not ip_block:
                raise RuntimeError(
                    'Unclassified response; rotation disabled'
                )

        if index + 1 < len(candidates):
            time.sleep(BASE_PAUSE + random.random())

    raise RuntimeError('Confirmed block; retry budget exhausted')


if __name__ == '__main__':
    body = fetch()
    print(f'received_bytes={len(body)}')

Run it with:

python rotate_on_block.py

Expected result: a successful request returns its body. A confirmed IP-specific rejection advances to another configured endpoint, within the attempt budget. Other failures stop immediately.

The loop creates a fresh Requests session for each attempt and disables inherited environment settings with trust_env = False. That keeps ambient proxy settings from changing the selected route. If your environment depends on a custom certificate bundle, configure that explicitly rather than disabling TLS verification.

The example also disables redirects. Inspect a redirect before following it, especially if it changes the destination host. A redirect is not an IP-block signal.

This design's advantage is predictable routing and a small retry budget. Its limitation is equally important: it depends on a valid rejection marker and correctly configured exits. It cannot infer why an arbitrary website rejected a request.

Verification configuration

Test the stopping conditions

Use an authorized staging target or a controlled test fixture. Do not generate deliberate failures against an unrelated production website.

1. Return a normal successful response and verify that the program makes only one attempt.
2. Return the documented IP-block response, then success, and verify that the program selects the next endpoint.
3. Return an ordinary forbidden response without the marker and verify that the program stops.
4. Return a rate-limit response with retry instructions and verify that the program does not rotate.
5. Return the confirmed block on every attempt and verify that the retry budget ends the operation.

Expected result: your logs show the intended branch without exposing credentials or full response bodies.

Record the target host, attempt index, status, classifier decision and elapsed time in production. Avoid recording session tokens or sensitive query parameters. Keep transport failures separate from target rejections so your team does not mistake a connection problem for blocked traffic.

Before rolling out this workflow, verify exit addresses using a destination you control or an approved diagnostic service. Log a safe exit identifier if your data policy allows it. A change in endpoint index is not enough to validate rotation.

Rotate when a scheduled check detects an exit rejection

The adjacent workflow is proactive testing before a scheduled collection job. Use an approved health-check resource that represents the integration, then choose an eligible endpoint before starting the batch.

Best for scheduled data collection: a preflight check catches connection and authentication failures before the batch begins. The limitation is that a successful check does not guarantee that every resource will accept the subsequent requests.

Store the result with its target and check time. Do not declare an exit globally blocked because one host refused it. Restrictions are destination-specific, and a stale success result does not prove current access.

If the preflight receives a rate limit, postpone the job according to the target's instructions. If it receives an unclassified refusal, send it for investigation. Keep the same narrow rotation rule as the request-time workflow.

Troubleshooting

The endpoint changes, but the exit stays the same

Check session assignment and sticky routing. Use your provider's documented rotation controls, then verify the exit independently. Closing a local connection does not force the provider to assign a different address.

Every proxy receives the same forbidden response

Stop after the configured budget. Check authorization, required headers, account permissions and the rejection marker with the target owner. Repeated refusal across exits is not evidence that a larger proxy list will solve the problem.

Proxy authentication fails before the target responds

Check the endpoint protocol, port and credentials. If authentication details contain reserved URL characters, encode them correctly in the connection string. Fix authentication before testing rotation.

The script works locally but fails in your job runner

Check outbound network rules, secret injection and certificate configuration. The script disables inherited Requests environment settings, so dependencies on an ambient proxy or certificate bundle need explicit configuration.

Successful responses contain the wrong content

Validate the expected schema, content type and required fields. Treat an unexpected login page or challenge as an application failure, not as usable data. Do not broaden the IP-block marker to absorb unrelated failures.

Customize your workflow

Keep a retry budget per job, not just per request. A small per-request limit still multiplies traffic when many workers fail simultaneously.

Add a shared pause for target-wide rate limits and a queue-level stop for repeated unclassified refusals. Separate safe reads from writes. A timed-out write can already have reached the destination, so retrying it requires application-specific idempotency controls.

Track accepted records separately from HTTP successes. The useful outcome is valid data collected within the target's access rules, not the number of addresses attempted.

FAQ

Why is my IP getting blocked, and does proxy rotation fix it?

Proxy rotation addresses a confirmed IP-specific rejection, not every blocked request. Diagnose authentication, rate limits, access permissions and response content before changing exits.

Should I rotate proxies after every HTTP 403 response?

No. Rotate only when an authorized integration has a documented signal identifying an IP-specific rejection; an ordinary HTTP 403 does not establish that cause.

Should I use another proxy after HTTP 429?

No. Stop the request loop and schedule work according to the target's rate-limit instructions, including Retry-After when supplied.

Does a new proxy connection always use a new IP address?

No. Exit assignment depends on the provider's routing and session rules. Verify the observed exit rather than assuming a new connection changed it.

Can I keep cookies when switching proxy IPs?

Only when the authorized integration supports that behavior. This example intentionally does not transfer cookies between attempts, so it is not a drop-in replacement for an authenticated browser session.

How many retries should this workflow make?

The example permits 3 total attempts and uses each configured endpoint at most once per operation. That is a sample safety limit, not a universal recommendation for every target.

What should I log for a proxy-rotation workflow?

Log the target host, attempt index, HTTP status, classification decision and elapsed time. Exclude credentials, session tokens and sensitive response bodies.

One last thing

Changing the exit is an action. Confirming the cause is the decision. Preserve that order in code.

Before adding concurrency, deliberately test an ordinary forbidden response without your approved marker. If the program rotates anyway, the classifier is too broad. Fix that branch before increasing the workload.

Related guides

• Proxy providers for Python developers
• SOCKS5 proxies for aiohttp
• Proxies for web scraping agencies
• Node.js proxy rotation with bounded dispatchers and worker limits