Proxy Rotation API vs Gateway: Checkpoints and Recovery

A gateway carries requests; a rotation API changes configuration. Route through the gateway and keep recovery in your application.

A gateway carries requests to the target. A rotation API changes the proxy assignment or configuration. Use each for its documented role, and keep retry and recovery logic in your application.

TL;DR

Gateways route scraping traffic; control APIs change configuration or request rotation when supported.
Node4 provides HTTP(S) and SOCKS5 gateways; your application controls the job queue, retries and checkpoints.
Proxy rotation does not provide zero downtime. Durable jobs, bounded retries, and checkpointed results prevent lost work.
Keep session identity separate from gateway configuration so endpoint changes do not silently break authenticated workflows.

Why this matters

A failed request and a failed pipeline are different problems. A proxy connection can fail while your queue retains the job. A successful response can still produce a broken dataset if your scraper saves a challenge page as valid content.

Treat zero downtime as a recovery objective, not a property of rotation. Your pipeline needs a durable record of unfinished work, a validation rule for responses, and a controlled way to replace connection settings.

For the underlying exit-selection behavior, start with the rotating proxies for web scraping guide; see Node4's rotating proxies and its pricing if you want that exit selection handled by the gateway rather than built by hand. This workflow focuses on application recovery, not an undocumented provider API.

Before you start

Have working proxy credentials, the documented endpoint and protocol, and permission to collect the target data. Confirm whether your provider rotates per request, per connection, or through a session setting.
Install Python and requests. Use a writable local directory for the example's SQLite database and response files. Keep credentials outside source control.
Check session scope before enabling rotation. Cookies, exit identity, and connection reuse are separate state. A workflow that authenticates on one exit can fail when its next request uses another.

The examples use an HTTP proxy endpoint for HTTPS requests. They do not demonstrate SOCKS5 configuration or implement a provider-specific rotation call. The gateway must already have the rotation behavior you need.

Gateway routing and API control

In this guide, a rotation API means a control endpoint that changes proxy state or requests a new assignment. A gateway means the proxy endpoint through which your application sends target requests. Some services use “API” to describe a fetch endpoint instead; identify which meaning applies before integrating.

Configuration unit	Best for	Benefit	Limitation
Gateway routing	Sending scraper requests through an established proxy endpoint	Keeps proxy connection settings separate from parsing logic	Exit-selection behavior depends on the provider's configuration
Rotation API control	Explicit, documented changes to proxy assignments or settings	Gives your application a control action outside the request path	Adds authentication, error handling, and coordination requirements

Use a gateway for transport and an API for control when both exist. Neither replaces checkpointing. An API call that requests rotation is not proof that the next request used a different exit.

1. Connection baseline

Establish a working request before adding retries or configuration reloads. Otherwise, repeated attempts hide a permanent authentication or protocol error.

Set PROXY_URL to the complete endpoint supplied by your provider, including its required authentication format.
Set TARGET_URL to an HTTPS resource you are authorized to fetch.
Run this connection test:

curl --proxy "$PROXY_URL" \
  --connect-timeout 10 \
  --max-time 30 \
  --fail-with-body \
  --output baseline-response.txt \
  "$TARGET_URL"

The 10-second connection timeout and 30-second total timeout are example operating limits, not provider performance figures. Change them to match your target and job deadline. Do not remove the deadline to accommodate a failing route.

Inspect baseline-response.txt. Confirm the content belongs to the requested resource, not a login screen or challenge page. An HTTP success status alone does not establish that scraping succeeded.

This command places the proxy URL in command arguments. On a shared runner, use your secret-management system and a protected curl configuration instead. Do not paste verbose connection logs into tickets without removing credentials.

Expected result: one verified response through the configured proxy. Record whether authentication, transport, and content validation each passed before continuing.

2. Durable job configuration

In this example, a job is one URL and its saved response. The URL is the deduplication key. That is appropriate for a single collection pass, not recurring observations of a changing page.

Create a protected working directory.
Store the proxy endpoint in proxy-url.txt with access restricted to the worker account.
Save the following script as worker.py and install its dependency:

python -m pip install requests

The worker retries selected transient failures, writes the response before acknowledging the job, and leaves unsuccessful jobs pending. It reloads the proxy configuration at the beginning of each job.

import hashlib
import os
import sqlite3
import time
from pathlib import Path

import requests

CONFIG = Path('proxy-url.txt')
OUTPUT = Path('responses')
OUTPUT.mkdir(exist_ok=True)
DB = sqlite3.connect('jobs.sqlite3')
DB.execute('''CREATE TABLE IF NOT EXISTS jobs (
    url TEXT PRIMARY KEY,
    done INTEGER NOT NULL DEFAULT 0
)''')

url = os.environ['TARGET_URL']
DB.execute('INSERT OR IGNORE INTO jobs(url) VALUES (?)', (url,))
DB.commit()


def fetch(url, proxy):
    with requests.Session() as session:
        session.trust_env = False
        session.proxies = {'http': proxy, 'https': proxy}
        for attempt in range(3):
            try:
                response = session.get(url, timeout=(10, 30))
                if response.status_code in (502, 503, 504):
                    if attempt == 2:
                        response.raise_for_status()
                else:
                    response.raise_for_status()
                    marker = os.environ.get('RESPONSE_MARKER')
                    if marker and marker not in response.text:
                        raise ValueError('Response validation failed')
                    return response.content
            except requests.exceptions.SSLError:
                raise
            except (requests.exceptions.Timeout,
                    requests.exceptions.ConnectionError):
                if attempt == 2:
                    raise
            time.sleep(2 ** attempt)


pending = DB.execute('SELECT url FROM jobs WHERE done = 0').fetchall()
for (url,) in pending:
    try:
        proxy = CONFIG.read_text().strip()
        if not proxy:
            raise ValueError('Empty proxy configuration')
        content = fetch(url, proxy)
        job_id = hashlib.sha256(url.encode()).hexdigest()
        destination = OUTPUT / f'{job_id}.bin'
        temporary = OUTPUT / f'{job_id}.tmp'
        with temporary.open('wb') as output:
            output.write(content)
            output.flush()
            os.fsync(output.fileno())
        temporary.replace(destination)
        DB.execute('UPDATE jobs SET done = 1 WHERE url = ?', (url,))
        DB.commit()
        print(f'completed {job_id}')
    except Exception as error:
        print(f'pending: {type(error).__name__}')

DB.close()

The worker allows 3 attempts per job, with waits of 1 second and 2 seconds between attempts. These are explicit example settings, not measured reliability claims. In requests, the 30-second read timeout limits waiting between received bytes; it is not a total job deadline.

Set RESPONSE_MARKER to text expected in the target response when that check fits your resource. A marker is a basic guard, not a replacement for schema validation. Add checks for required fields before marking production jobs complete.

Expected result: valid responses become files, completed jobs remain acknowledged, and failures remain available for another run.

3. Recovery checks

The worker separates retrieval from acknowledgment. A crash before the database update leaves a job pending, even if its response file already exists. Reprocessing overwrites the same destination instead of creating another filename.

Run python worker.py against your authorized test resource.
Run it again with the same TARGET_URL. The completed URL should not be fetched again.
Add a different test URL, then interrupt the process before completion. Restart and confirm unfinished work remains pending.
Test response validation deliberately. Set a marker absent from the page and confirm the worker does not acknowledge that response.

This is an at-least-once retrieval pattern, not an exactly-once pipeline. A request can reach the target even when your application receives no response. Retry read-only retrievals; do not apply the same policy to purchases, account changes, or other side effects.

The example has important limits. It runs sequentially, buffers the response in memory, and does not coordinate multiple workers. Atomic file replacement prevents readers from seeing a partially written replacement, but this script does not establish full power-loss durability across every filesystem.

For a distributed deployment, use explicit job claims, lease expiry, and a storage system suited to your environment. For recurring collection, include an observation identifier in the job key rather than permanently deduplicating by URL.

Expected result: process restarts do not discard unfinished jobs, and repeated execution does not create duplicate response filenames.

4. Configuration replacement

A worker should not read a configuration file while another process is halfway through writing it. Stage the replacement beside the active file, validate it, then replace the active path.

Export NEXT_PROXY_URL with the replacement endpoint from your provider.
Test the replacement using the baseline curl command, substituting NEXT_PROXY_URL for PROXY_URL.
After the test succeeds, replace the configuration:

python - <<'PY'
import os
from pathlib import Path

active = Path('proxy-url.txt')
staged = active.with_name(active.name + '.next')
with staged.open('w') as output:
    os.chmod(staged, 0o600)
    output.write(os.environ['NEXT_PROXY_URL'] + '\n')
    output.flush()
    os.fsync(output.fileno())
os.replace(staged, active)
PY

Submit a new authorized test job. Confirm it completes using the replacement configuration before retiring the old endpoint.

On a POSIX system, 0o600 restricts file access to its owner. Use equivalent access controls on other operating systems. Keep the staged and active files on the same filesystem for atomic replacement.

The sample worker reads configuration between jobs. It does not switch a running request or rescue an invalidated connection. Keep old credentials usable while in-flight work drains when your provider supports that transition.

Expected result: later jobs receive the replacement configuration without reading a partially written file. Test restart behavior separately from configuration replacement in your deployment checklist.

Variant: API-triggered rotation between sessions

Use explicit rotation only when the provider documents the control call and its scope. Do not invent an endpoint, username suffix, or session parameter from another provider's example.

Finish or checkpoint the logical session that owns the current exit.
Call the documented control endpoint with its required authentication.
Check the response according to the provider's contract. Keep control failures separate from target-request failures.
Close the scraper's existing transport session if the documented behavior requires a new connection.
Verify the resulting exit through an authorized diagnostic service, then start the next logical session.

Best for: workflows that need an explicit boundary between exit assignments. The benefit is deliberate coordination; the cost is another failure path and tighter coupling to provider behavior.

Never let one worker rotate a shared assignment without coordinating its peers. In your session design, define who owns the assignment and what happens to outstanding requests. Repeatedly calling a rotation API after every target error creates a second retry loop, not a recovery policy.

Troubleshooting

Proxy authentication fails

An HTTP 407 means proxy authentication is required. Check credential formatting, the endpoint's authentication method, and any documented source-address restrictions. The example leaves this job pending rather than retrying the same rejected credentials.

The target returns a rate-limit response

An HTTP 429 is a signal to slow down, not automatically rotate. Pause affected jobs, respect Retry-After when supplied, and reduce request pressure. The sample does not automatically retry this status; add target-aware scheduling before production use.

Rotation breaks an authenticated session

Keep related requests inside the provider's documented session mechanism. Preserve application cookies separately, and do not assume a persistent TCP connection keeps the same exit. Restart the logical session when its identity is no longer valid.

A TLS certificate error repeats

Check the target hostname, trust store, and any intentional interception configuration. Do not disable certificate verification to make scraping continue. The example treats certificate errors as configuration failures instead of transient retries.

Jobs complete with unusable content

Validate the response before writing completion state. Check required fields, expected page identity, and parser output. Retain a sanitized diagnostic sample so you can distinguish a target layout change from a proxy connection problem.

Customize your workflow

The next improvement is observability, not more rotation. Record job state, attempt count, status code, validation outcome, configuration version, and elapsed time. Never record proxy passwords or authenticated endpoint strings.

Separate transport errors from target rejections and parser failures. Each category needs a different response. Increase concurrency only after adding coordinated job ownership and a target-specific request budget.

Use the rotating proxy API guide for developers when evaluating control interfaces. Require documented session scope, authentication, and configuration-change behavior before integrating any control call.

FAQ

What's the difference between a proxy rotation API and a gateway?

A gateway carries target requests through a proxy; a rotation control API changes settings or requests an assignment change. Some providers call a fetch service an API, so confirm the documented meaning before choosing an integration.

Is a gateway better than an API for a scraping pipeline?

A gateway is the direct choice when your scraper already handles fetching and parsing. A control API serves a separate purpose when you need documented assignment or configuration actions.

Does proxy rotation eliminate downtime?

No, proxy rotation does not eliminate downtime on its own. Durable jobs, bounded retries, response validation, and controlled configuration changes keep connection failures from becoming lost work.

Should I rotate the proxy after every failed request?

No, classify the failure before changing the exit. Authentication errors require configuration fixes, rate limits require slower scheduling, and session-sensitive workflows require identity continuity.

Does the Python example rotate an IP itself?

No, the example sends traffic through the configured proxy endpoint. Exit selection remains the provider's documented behavior; the script adds retries, durable job state, and configuration reloads.

Can multiple workers share the example's job database?

The example does not implement safe multi-worker job claiming. Add transactional claims or a queue with leases before running concurrent workers against the same pending jobs.

How do I avoid duplicate results after a worker restart?

Use a stable job key and write the result before acknowledging completion. The example uses the URL as its key; recurring collection needs an observation identifier so later snapshots are not skipped.

One last thing

Test recovery by interrupting a worker after it saves a response but before it acknowledges the job. That boundary exposes duplicate-handling mistakes that a successful connection test never exercises. A pipeline earns its continuity from recoverable state, not from the number of exits it can request.