Node.js Proxy Rotation: Dispatchers, Workers and Retries

Build a Node.js proxy setup with bounded rotation, explicit dispatchers, timeouts and safe retries. Verify exit behavior before deployment.

Use an explicit proxy dispatcher in Node.js, limit concurrent requests, and retry only failures that can safely be retried. This workflow builds a runnable HTTP-proxy client that way, without treating rotation as permission to retry every failure.

TL;DR

Why this matters

A proxy changes the network path. It does not make a blocked request successful, preserve application state, or tell you whether a retry is safe. Those decisions belong in your client.

The common failure is a loop that changes credentials, retries immediately, and sends more traffic to an already unhappy destination. Rotate deliberately, limit active requests, and classify failures before retrying. That rule applies to a developer's script, an agency's collection job, and a data team's scheduled pipeline.

For the network-side decisions, read the rotating proxies for web scraping guide; if you need a gateway with rotation built in rather than building this dispatcher yourself, see Node4's rotating proxies and its pricing. The implementation below handles the application side: dispatch, response consumption, worker limits, and shutdown.

Before deployment, test the actual package and runtime combination you ship. A working development request is not proof that your container, credentials, and production network behave identically.

Before you start

The non-obvious trap is mixing connection management with identity management. Closing an agent after every request is not a documented way to select a new exit. Use the endpoint's documented rotation or session controls instead.

This guide does not configure SOCKS5. Undici's ProxyAgent handles HTTP-style proxy connections; do not pass a SOCKS5 URL into this example and expect it to work.

Proxy configuration

  1. Create the project and install Undici:
npm init -y
npm install undici
  1. Supply your endpoint and destination through environment variables:
read -r -s -p 'Proxy URL: ' PROXY_URL
printf '\n'
export PROXY_URL
read -r -p 'Authorized HTTPS target URL: ' TARGET_URL
export TARGET_URL
  1. Confirm that the proxy URL uses http: or https:. If authentication is required, include the supplied username and password using URL encoding for reserved characters.

Do not put credentials into a committed configuration file. Avoid verbose HTTP traces in shared logs, and remember that environment variables still need access controls on the machine running the process.

Expected result: the process has a proxy endpoint and an authorized destination, but no requests have been sent yet.

Commit package-lock.json and use npm ci in deployment. Record the runtime and dependency versions with the build so you can reproduce a connection failure rather than debugging against a different install.

Request dispatcher

  1. Save the following as proxy-worker.mjs.
  2. Keep the limits explicit. The example uses 4 workers, a 15,000-millisecond timeout, 3 attempts, and a 1 MiB response limit. These are example safety limits, not provider specifications or measured recommendations.
  3. Run it against a small response first.
import { fetch, ProxyAgent } from 'undici';
import { setTimeout as sleep } from 'node:timers/promises';

const WORKERS = 4;
const TIMEOUT_MS = 15_000;
const ATTEMPTS = 3;
const MAX_BYTES = 1_048_576;

const target = new URL(process.env.TARGET_URL);
if (target.protocol !== 'https:') {
  throw new Error('TARGET_URL must use HTTPS');
}

const rawProxies = process.env.PROXY_URLS
  ? JSON.parse(process.env.PROXY_URLS)
  : [process.env.PROXY_URL];
if (!Array.isArray(rawProxies) || !rawProxies.length) {
  throw new Error('Provide at least one proxy');
}

const agents = rawProxies.map((raw) => {
  const proxy = new URL(raw);
  if (!['http:', 'https:'].includes(proxy.protocol)) {
    throw new Error('Only HTTP-style proxies are supported');
  }
  const user = decodeURIComponent(proxy.username);
  const password = decodeURIComponent(proxy.password);
  proxy.username = '';
  proxy.password = '';
  const options = { uri: proxy.href };
  if (user || password) {
    options.token = `Basic ${Buffer.from(
      `${user}:${password}`
    ).toString('base64')}`;
  }
  return new ProxyAgent(options);
});

const jobs = process.argv.slice(2).map((value) => {
  const url = new URL(value, target);
  if (url.origin !== target.origin) {
    throw new Error('Job URL must match TARGET_URL origin');
  }
  return url.href;
});
if (!jobs.length) jobs.push(target.href);

const retryStatuses = new Set([502, 503, 504]);
const retryCodes = new Set([
  'ECONNRESET', 'ETIMEDOUT', 'UND_ERR_CONNECT_TIMEOUT',
  'UND_ERR_SOCKET',
]);
let cursor = 0;
let dispatchIndex = 0;

async function request(url) {
  for (let attempt = 1; attempt <= ATTEMPTS; attempt++) {
    const dispatcher = agents[dispatchIndex++ % agents.length];
    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), TIMEOUT_MS);
    const started = Date.now();
    try {
      const response = await fetch(url, {
        dispatcher,
        signal: controller.signal,
        redirect: 'error',
      });
      if (!response.ok) {
        const retryAfter = response.headers.get('retry-after');
        await response.body?.cancel();
        if (retryAfter !== null) {
          throw new Error(`Defer HTTP ${response.status}; Retry-After=${retryAfter}`);
        }
        if (retryStatuses.has(response.status) && attempt < ATTEMPTS) {
          clearTimeout(timer);
          await sleep(500 * attempt + Math.floor(Math.random() * 250));
          continue;
        }
        throw new Error(`HTTP ${response.status}`);
      }
      let bytes = 0;
      for await (const chunk of response.body ?? []) {
        bytes += chunk.byteLength;
        if (bytes > MAX_BYTES) {
          controller.abort();
          throw new Error('Response exceeds configured byte limit');
        }
      }
      return { status: response.status, bytes, attempt,
        durationMs: Date.now() - started };
    } catch (error) {
      const code = error.cause?.code ?? error.code;
      if (!retryCodes.has(code) || attempt === ATTEMPTS) throw error;
      clearTimeout(timer);
      await sleep(500 * attempt + Math.floor(Math.random() * 250));
    } finally {
      clearTimeout(timer);
    }
  }
}

async function worker() {
  while (cursor < jobs.length) {
    const index = cursor++;
    try {
      console.log(JSON.stringify({ job: index, ...await request(jobs[index]) }));
    } catch (error) {
      process.exitCode = 1;
      console.error(JSON.stringify({ job: index, error: error.message }));
    }
  }
}

try {
  await Promise.all(Array.from(
    { length: Math.min(WORKERS, jobs.length) }, worker
  ));
} finally {
  await Promise.all(agents.map((agent) => agent.close()));
}
node proxy-worker.mjs

The dispatcher routes each request through the selected endpoint. The client consumes the response before freeing its worker, limits decoded response bytes, and keeps the timeout active during body consumption. It rejects redirects rather than following requests outside the approved origin.

Expected result: a successful run prints a JSON record containing the HTTP status, bytes consumed, attempt number, and request duration. Failure produces an error record and a nonzero process exit status.

This example validates routing mechanics, not extraction. It discards the body after counting bytes. Add your parser inside the size-limited consumption path when you need collected data.

Rotation and worker controls

  1. Start with one supplied proxy endpoint. If that endpoint rotates exits server-side, keep using it according to its documented behavior.
  2. To cycle through multiple supplied endpoints, set PROXY_URLS to a JSON array of complete proxy URLs through your secret mechanism. The array overrides PROXY_URL.
  3. Submit several same-origin job paths:
node proxy-worker.mjs "$TARGET_URL" "$TARGET_URL" "$TARGET_URL"
  1. Adjust WORKERS only after checking destination limits and the proxy account's connection allowance.

The array selects endpoints in round-robin order, including retry attempts. Endpoint rotation does not prove exit-IP rotation. Multiple endpoints can use the same exit, and a single gateway can select different exits according to its configuration.

A worker limit is also not a requests-per-second limit. Short responses let workers issue new requests quickly. Add a destination-specific rate limiter when your authorized workload requires spacing between requests.

The retry policy is intentionally narrow. It retries selected transport errors and HTTP 502, 503, or 504 responses without Retry-After. It does not retry HTTP 403, authentication failures, oversized responses, or request aborts. The short backoff values are demonstration settings, not a throughput target.

Expected result: no more than the configured worker count processes jobs simultaneously, and retries stop at the configured attempt limit. The process closes dispatchers after all workers finish.

Exit verification and release checks

  1. Point TARGET_URL at an HTTPS endpoint you control that records the connecting address server-side. Do not trust an application-supplied forwarding header as proof of the network exit.
  2. Compare a direct request with proxied requests using the server's connection records. Inspect the successful response separately with curl when needed:
curl --proxy "$PROXY_URL" --max-time 15 "$TARGET_URL"
  1. Repeat requests according to the gateway's documented rotation trigger. Test from the production environment, not only your laptop.
  2. Exercise a slow response, a denied response, and a temporary server failure on your controlled endpoint.

Confirm that timeouts release workers, denied responses stop, and retryable failures respect the attempt limit. Check that proxy credentials, authorization headers, and destination query strings do not appear in shared logs.

Expected result: you can distinguish a successful proxy connection from a successful data collection job. Save the test configuration and observations with the deployment record, without secrets.

The example's same-origin check is not a complete server-side request forgery defense. Keep job destinations under operator control. A service accepting customer-supplied URLs needs separate hostname, address, and network-egress controls.

Keep one exit for a stateful workflow

Use a sticky session when several authorized requests must share a network identity. A login, paginated session, or stateful application flow should not automatically receive a different identity on each retry.

Assign the documented session configuration at the job boundary. Keep that configuration unchanged through the workflow, and maintain cookies separately. A proxy dispatcher is not a browser session or a cookie jar.

On Node4's rotating gateway, a sticky session id pins one exit for up to 30 minutes from first assignment, and a later request does not restart that clock. Residential sessions are best effort, with a 30-minute ceiling: the exit can drop sooner if its device goes offline, so switch to a new session ID after repeated failures rather than retrying the same one. Set the workflow's deadline inside whichever window applies to your gateway rather than extending work indefinitely because requests still succeed.

The rotating variant suits independent requests; the sticky variant suits identity-dependent requests on that same rotating gateway. Rotation spreads network identities but complicates state. Sticky routing preserves identity for its supported window but requires expiry handling.

Do not send a partially completed stateful workflow through the example's global endpoint rotation unchanged. Give that workflow a fixed dispatcher selection and a separate retry policy. GET is not a guarantee that the surrounding business operation is safe to repeat.

Troubleshooting

Customize your workflow

Replace the in-memory job list with a durable queue when jobs must survive process restarts. Record successful completion only after parsing and storage succeed. Otherwise, a healthy HTTP response can conceal a failed collection job.

Add a scheduler for deferred responses. Parse Retry-After as either delay seconds or an HTTP date, and release the worker until the scheduled retry. Persist that decision rather than sleeping indefinitely inside a busy worker.

Track outcomes by destination and job type. Useful fields include status, bytes, attempt count, elapsed time, and parser result. Keep credentials and sensitive URLs out of telemetry.

Separate network success from valid-data success and queue completion. Increase concurrency only when these signals remain acceptable under your approved workload. The sample's worker count is a starting configuration, not evidence of capacity.

FAQ

What's the simplest Node.js proxy setup for HTTP requests?

Use an explicit proxy dispatcher, such as Undici ProxyAgent, with your supplied HTTP-style endpoint. Add timeouts and response limits before scaling beyond a connectivity test.

Does setting a proxy environment variable automatically route Node.js requests?

This workflow routes requests through an explicitly configured dispatcher. Do not assume an environment variable is honored unless the exact runtime and HTTP client document that behavior.

Does a rotating proxy change the IP on every request?

A rotating proxy follows its configured rotation policy, not a universal per-request rule. Verify the endpoint's documented trigger against a controlled destination's connection records.

Can I use SOCKS5 with this code?

No, this example uses Undici ProxyAgent for HTTP-style proxies. A SOCKS5 endpoint needs a compatible client or transport integration.

Should I rotate proxies when I receive HTTP 403?

No, HTTP 403 is not automatically a retryable network failure. Check authorization and destination restrictions instead of cycling identities to bypass the response.

How do I keep the same proxy identity for several requests?

Use the provider's documented sticky-session configuration and keep it stable for the workflow. Maintain application cookies separately and stop before the documented session window expires.

Is this worker script a complete production scraping system?

No, it provides bounded proxy dispatch, limited retries, response-size protection, and shutdown. Production collection also needs destination approval, durable scheduling, parsing, storage, and operational monitoring.

One last thing

Test the failure path before increasing the worker count. A fast success proves little about shutdown, deferred jobs, or oversized responses. Make your controlled endpoint fail deliberately, then confirm that the client stops, releases resources, and reports the right outcome without exposing credentials.

Related guides