Selenium Rotating Proxy: Auth, Rotation and Leak Checks

Selenium cannot rotate and Chromium drops proxy passwords. Three ways to rotate, four ways to authenticate, and the check that proves the exit changed.

Two facts explain nearly every Selenium proxy problem, and neither is a bug in your code. First, Selenium has no rotation primitive: a WebDriver session is one browser process, and that process takes exactly one proxy configuration, fixed at launch. Second, Chromium ignores credentials in --proxy-server. Put http://user:pass@host:port in the flag and the browser connects to the proxy with no credentials at all, receives a 407, and shows an authentication dialog that no find_element call can reach because it is drawn outside the page.

Everything else in this guide follows from those two facts. Rotation has to happen either around the browser (restart it), beside it (a local forwarder) or behind it (a gateway that rotates for you). Authentication has to happen through something other than the flag. The article works through each option with runnable code, then covers the leaks a browser adds that a plain HTTP client never had, and ends with the errors you will actually see.

It is Selenium-specific. Plain requests or httpx sessions are a different and much easier job, covered in Python requests proxy setup. Puppeteer has the same Chromium constraint with different answers, in Puppeteer rotating proxies.

Where a proxy attaches in Selenium

For an endpoint that needs no credentials, the setup is one option per browser. Chrome, Chromium and Edge take a launch flag:

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument("--proxy-server=http://HOST:PORT")
driver = webdriver.Chrome(options=options)

driver.get("https://api.ipify.org/")
print(driver.find_element("tag name", "body").text)
driver.quit()

The flag accepts socks5:// as well as http://; there is no separate https:// proxy to configure, because HTTPS targets tunnel through the same HTTP proxy with a CONNECT request. Firefox does not read the flag and uses preference keys instead:

options = webdriver.FirefoxOptions()
options.set_preference("network.proxy.type", 1)
options.set_preference("network.proxy.http", "HOST")
options.set_preference("network.proxy.http_port", PORT)
options.set_preference("network.proxy.ssl", "HOST")
options.set_preference("network.proxy.ssl_port", PORT)
driver = webdriver.Firefox(options=options)

Set both the http and ssl pairs. A Firefox profile with only the http pair proxies plain-HTTP navigations and sends every https:// page directly from your own address, and nothing warns you.

The IP echo at the end of the first example is not decoration. A proxy setting that did not take does not raise; the browser simply browses from your machine. Every configuration change in this article should end with that same check.

Why the password disappears, and the four ways around it

Chromium's proxy flag carries a server, not an identity. When the proxy answers 407, the browser expects a human to type into a dialog. In automation there is no human, so you need one of four arrangements, and which one fits depends on your language, your CI setup and whether the target needs a specific exit.

Whitelist your source address and send no credentials. Register the public IP of the machine running Selenium on the Auth & IP Access page and the proxy stops asking. The browser launches with the plain --proxy-server flag and there is nothing to inject. This is the arrangement to prefer wherever your egress is stable, which describes servers and self-hosted runners; hosted CI runners draw a new address per job and need one of the other three. The scope differs by product: on static datacenter proxies nothing is lost, on the rotating gateways you can still keep country and session targeting by sending the username with an empty password, and on the residential gateway a whitelisted connection sends nothing at all, so it rotates freely and cannot be pinned to a country or session. If a residential browser job needs a specific exit, keep credentials. IP whitelisting best practices covers the operational side, including what happens when the runner's address changes under you.

In Java, register the credentials with the driver. Selenium 4's HasAuthentication answers the proxy's challenge through DevTools, with no extension and no forwarder:

import org.openqa.selenium.HasAuthentication;
import org.openqa.selenium.UsernameAndPassword;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;

ChromeOptions options = new ChromeOptions();
options.addArguments("--proxy-server=http://HOST:PORT");
ChromeDriver driver = new ChromeDriver(options);

((HasAuthentication) driver).register(UsernameAndPassword.of("USERNAME", "PASSWORD"));
driver.get("https://api.ipify.org/");

It rides on the Chrome DevTools Protocol, so it works with a local Chrome or Edge driver (a RemoteWebDriver needs wrapping with Augmenter first), and the selenium-devtools artifact on your classpath has to match the browser's major version. Firefox is not covered.

In Python, generate a small extension that supplies the credentials. Python gained a driver-level answer late: Selenium 4.46 added driver.network.add_auth_handler("USERNAME", "PASSWORD") over BiDi, which needs options.enable_bidi = True before the driver starts and a browser whose BiDi implementation forwards the proxy's challenge to it, so end with the IP echo rather than assuming it did. On anything older, the route that works on every Selenium 4 release is a Manifest V3 extension, written to disk at runtime, that sets the proxy and answers onAuthRequired:

import json
import zipfile
from string import Template

from selenium import webdriver

WORKER = Template("""
chrome.proxy.settings.set({
  value: {
    mode: "fixed_servers",
    rules: {
      singleProxy: { scheme: "http", host: "$host", port: $port },
      bypassList: ["localhost", "127.0.0.1"]
    }
  },
  scope: "regular"
});
chrome.webRequest.onAuthRequired.addListener(
  (details, callback) => {
    callback({ authCredentials: { username: "$user", password: "$password" } });
  },
  { urls: ["<all_urls>"] },
  ["asyncBlocking"]
);
""")

MANIFEST = {
    "manifest_version": 3,
    "name": "proxy auth",
    "version": "1.0",
    "permissions": ["proxy", "webRequest", "webRequestAuthProvider"],
    "host_permissions": ["<all_urls>"],
    "background": {"service_worker": "worker.js"},
}

def proxy_extension(host, port, user, password, path="proxy_auth.zip"):
    with zipfile.ZipFile(path, "w") as z:
        z.writestr("manifest.json", json.dumps(MANIFEST))
        z.writestr("worker.js", WORKER.substitute(
            host=host, port=port, user=user, password=password))
    return path

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
options.add_extension(proxy_extension(
    "gw-rotating_datacenter.node4.io", 8083, "USERNAME", "PASSWORD"))
driver = webdriver.Chrome(options=options)
driver.get("https://api.ipify.org/")

Three things break this in practice. The old headless mode cannot load extensions at all. Before Chrome 132 that is what a bare --headless meant; since 132 a bare --headless is the new mode and the old one only survives as the separate chrome-headless-shell binary, which still cannot load them. --headless=new means the same thing on every version, which is why the examples use it. Manifest V2 examples, which are most of what a search turns up, no longer load in current Chrome; the webRequestAuthProvider permission is what makes the blocking auth listener legal under V3. And because the proxy is now set by the extension, do not also pass --proxy-server; two sources of proxy configuration are one more than you can debug.

Run a local forwarder that adds the header for you. selenium-wire was the standard answer for years: it starts a local proxy in front of the browser, injects Proxy-Authorization on the way out, and lets you swap the upstream mid-session with driver.proxy = {...}. It still works for many people, but the project was archived in January 2024 and receives no fixes for new Chrome or Python releases. It also intercepts TLS, which breaks against targets that pin certificates. Treat it as a compatibility tool for an existing suite, not the foundation of a new one. A credential-free local forwarder (any small proxy that accepts unauthenticated connections from localhost and authenticates upstream) does the same job without the TLS interception, at the cost of one more process to supervise.

If none of these lands and the browser still shows a 407, confirm the credentials in curl before touching Selenium again; fixing 407 Proxy Authentication Required walks that isolation step.

Three ways to rotate in Selenium

Selenium will not rotate for you, so the decision is where the rotation lives.

Restart the driver per proxy. The simplest mechanism, and for a job of a few hundred pages the right one. Keep a cycle over the addresses from your dashboard and launch a fresh driver per batch:

import itertools
from selenium import webdriver

PROXIES = ["203.0.113.10:8080", "203.0.113.11:8080", "203.0.113.12:8080"]
cycle = itertools.cycle(PROXIES)

def driver_for(proxy):
    options = webdriver.ChromeOptions()
    options.add_argument("--headless=new")
    options.add_argument(f"--proxy-server=http://{proxy}")
    return webdriver.Chrome(options=options)

for batch in batches(urls, size=25):
    driver = driver_for(next(cycle))
    try:
        for url in batch:
            driver.get(url)
            handle(driver)
    finally:
        driver.quit()

The cost is a Chromium launch per rotation, which is hundreds of megabytes and a couple of seconds each time. The list above is static datacenter addresses with whitelisting on, so no auth code is needed; with credentials, generate the extension per proxy inside driver_for. The other cost is that the list is now yours to keep healthy: an address that times out stays in the cycle until you remove it.

Switch a local forwarder mid-session. With selenium-wire or your own forwarder in front of the browser, the upstream can change without restarting Chromium, so one long-lived driver walks a rotation. It is the only mechanism that gives you rotation and a persistent browser profile at once, which is useful when logging in is expensive. It inherits every caveat from the previous section.

Point the browser at a gateway and let the exit rotate behind it. One hostname, one launch, and the pool rotates for you: each new connection through gw-rotating_datacenter.node4.io (each HTTPS tunnel, or each plain-HTTP request) gets its own pick from the pool. No list to manage, no restarts, and pool health is the provider's problem. This is the mechanism for anything that runs unattended.

There is a wrinkle specific to browsers that the gateway does not cause but does expose. Rotation happens per connection, and a browser opens several of them to load one page, then keeps them alive for the next. A single page load through a rotating gateway can therefore fetch its document from one exit and its scripts from another, which is harmless on most sites and confusing on a site that binds a session to an address. The fix is to pin a session per driver: append a session segment to the username, and everything that driver loads shares one exit until you relaunch with a new id.

import uuid

def session_driver(user, password):
    session_id = uuid.uuid4().hex[:12]
    options = webdriver.ChromeOptions()
    options.add_argument("--headless=new")
    options.add_extension(proxy_extension(
        "gw-residential.node4.io", 8082,
        f"{user}-session-{session_id}", password))
    return webdriver.Chrome(options=options)

Rotation is now a relaunch with a fresh id, which is the driver-restart pattern again, except the addresses come from the pool rather than a list you maintain. Pinning is a first-class feature of the residential gateway. On the rotating gateways it depends on the gateway's session store: when that store is unreachable a pinned request is refused with a 503 rather than served from a rotating exit, and every rotating response carries an X-Node4-Sticky header whose none value means no session segment arrived (a misspelled segment key is refused with a 407 instead). Test it before a job depends on it. How long a pin holds and when you should let it go is in sticky versus rotating sessions.

Verify the exit actually changed

Rotation that did not happen looks identical to rotation that did, right up until a target blocks the one address every worker has been sharing. Make the check part of the job:

def exit_ip(driver):
    driver.get("https://api.ipify.org/")
    return driver.find_element("tag name", "body").text.strip()

a = session_driver("USERNAME", "PASSWORD")
b = session_driver("USERNAME", "PASSWORD")
assert exit_ip(a) != exit_ip(b), "two sessions, one exit: pinning or rotation is off"

Two drivers with different session ids should report different addresses; the same driver asked twice should report the same one. If the first assertion fails through a gateway, the session segment is being ignored or the plan does not include the gateway at all. If it fails with a proxy list, the list has duplicates or the flag did not take.

Where datacenter exits lose

A rotating datacenter pool solves per-address rate limits, and it does not solve classification. Targets that take bot traffic seriously look up the address's network and treat the whole hosting range as automation, so rotating within that range produces a different address with the same verdict. When a job that worked on a lenient site gets challenges or empty shells from a defended one, and the browser fingerprint is clean, that is usually what happened. The answer is a residential exit, whose address belongs to a consumer network, run with credentials so the session segment can hold it steady for the length of a flow.

Residential does not fix the other half. A headless browser announces itself through timing, missing plugins, the navigator.webdriver flag and a TLS handshake that does not match its claimed user agent, and a clean address under a flagged fingerprint still gets flagged. Tools like undetected-chromedriver address the fingerprint and do nothing for the proxy; treat the two as separate problems. Why your IP is getting blocked separates address reputation from fingerprint so you can tell which one you are looking at.

There is also the question of whether a browser is the right tool for the job at all. A browser fetches every script, image and font the page references, and through a metered plan that multiplier is on the bill. When the data is in the HTML or in a JSON endpoint, a Scrapy spider through the same gateway costs a fraction per page and rotates more cleanly; proxies for Scrapy covers that product choice. Save the browser for pages that genuinely need JavaScript to exist.

Leaks a browser adds

An HTTP client sends what you told it to. A browser resolves names, opens UDP sockets and prefetches, and only some of that follows the proxy setting.

• WebRTC discovers routes by sending UDP to STUN servers, outside the proxy, which lets a page's JavaScript learn your real address. Launch with --force-webrtc-ip-handling-policy=disable_non_proxied_udp on Chromium, or set media.peerconnection.enabled to false in Firefox.
• DNS through SOCKS5. With a socks5:// proxy Chromium hands the hostname to the proxy for the request itself, but prefetching and other lookups still run locally and reveal what you are about to visit; add --host-resolver-rules="MAP * ~NOTFOUND , EXCLUDE HOST" so every local lookup except the proxy's own name is refused and resolution happens at the exit. Firefox needs network.proxy.socks_remote_dns set to true. HTTP proxies do not have this problem, because the hostname travels inside the CONNECT request.
• The bypass list. Anything on --proxy-bypass-list (or Firefox's network.proxy.no_proxies_on) resolves and connects directly. Keep it to localhost.
• The profile. Timezone, language and cookies are the browser's own. An exit in one country and a profile claiming another is a signal on its own; align them per driver if the target cares.

Load a public WebRTC leak test page in the proxied browser after any change here. The only address on display should be the proxy exit.

The errors you will actually see

Work the table top to bottom, and in every case start from curl. If curl -x http://HOST:PORT https://api.ipify.org/ fails, Selenium was never the problem, and proxy connection refused or timing out is the guide to read next.

Choosing the mechanism

For a test suite that needs a stable identity from a known address, whitelist a static datacenter proxy and pass the flag; there is nothing else to build. For a scraping job that runs unattended, point the browser at a gateway, pin a session per driver, and relaunch to rotate, escalating from the rotating gateway to residential on evidence of classification rather than in advance. Keep selenium-wire for the suite that already depends on it. Plans and what each includes are on the pricing page, and proxies for browser automation lays the same three products side by side for a Selenium, Puppeteer or Playwright job.