Selenium Rotating Proxy: Auth, Rotation and Leak Checks
Selenium cannot rotate and Chromium drops proxy passwords. Three ways to rotate, four ways to authenticate, and the check that proves the exit changed.
Two facts explain nearly every Selenium proxy problem, and neither is a bug in your code. First, Selenium has no rotation primitive: a WebDriver session is one browser process, and that process takes exactly one proxy configuration, fixed at launch. Second, Chromium ignores credentials in --proxy-server. Put http://user:pass@host:port in the flag and the browser connects to the proxy with no credentials at all, receives a 407, and shows an authentication dialog that no find_element call can reach because it is drawn outside the page.
Everything else in this guide follows from those two facts. Rotation has to happen either around the browser (restart it), beside it (a local forwarder) or behind it (a gateway that rotates for you). Authentication has to happen through something other than the flag. The article works through each option with runnable code, then covers the leaks a browser adds that a plain HTTP client never had, and ends with the errors you will actually see.
It is Selenium-specific. Plain requests or httpx sessions are a different and much easier job, covered in Python requests proxy setup. Puppeteer has the same Chromium constraint with different answers, in Puppeteer rotating proxies.
Where a proxy attaches in Selenium
For an endpoint that needs no credentials, the setup is one option per browser. Chrome, Chromium and Edge take a launch flag:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--proxy-server=http://HOST:PORT")
driver = webdriver.Chrome(options=options)
driver.get("https://api.ipify.org/")
print(driver.find_element("tag name", "body").text)
driver.quit()The flag accepts socks5:// as well as http://; there is no separate https:// proxy to configure, because HTTPS targets tunnel through the same HTTP proxy with a CONNECT request. Firefox does not read the flag and uses preference keys instead:
options = webdriver.FirefoxOptions()
options.set_preference("network.proxy.type", 1)
options.set_preference("network.proxy.http", "HOST")
options.set_preference("network.proxy.http_port", PORT)
options.set_preference("network.proxy.ssl", "HOST")
options.set_preference("network.proxy.ssl_port", PORT)
driver = webdriver.Firefox(options=options)Set both the http and ssl pairs. A Firefox profile with only the http pair proxies plain-HTTP navigations and sends every https:// page directly from your own address, and nothing warns you.
The IP echo at the end of the first example is not decoration. A proxy setting that did not take does not raise; the browser simply browses from your machine. Every configuration change in this article should end with that same check.
Why the password disappears, and the four ways around it
Chromium's proxy flag carries a server, not an identity. When the proxy answers 407, the browser expects a human to type into a dialog. In automation there is no human, so you need one of four arrangements, and which one fits depends on your language, your CI setup and whether the target needs a specific exit.
Whitelist your source address and send no credentials. Register the public IP of the machine running Selenium on the Auth & IP Access page and the proxy stops asking. The browser launches with the plain --proxy-server flag and there is nothing to inject. This is the arrangement to prefer wherever your egress is stable, which describes servers and self-hosted runners; hosted CI runners draw a new address per job and need one of the other three. The scope differs by product: on static datacenter proxies nothing is lost, on the rotating gateways you can still keep country and session targeting by sending the username with an empty password, and on the residential gateway a whitelisted connection sends nothing at all, so it rotates freely and cannot be pinned to a country or session. If a residential browser job needs a specific exit, keep credentials. IP whitelisting best practices covers the operational side, including what happens when the runner's address changes under you.
In Java, register the credentials with the driver. Selenium 4's HasAuthentication answers the proxy's challenge through DevTools, with no extension and no forwarder:
import org.openqa.selenium.HasAuthentication;
import org.openqa.selenium.UsernameAndPassword;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
ChromeOptions options = new ChromeOptions();
options.addArguments("--proxy-server=http://HOST:PORT");
ChromeDriver driver = new ChromeDriver(options);
((HasAuthentication) driver).register(UsernameAndPassword.of("USERNAME", "PASSWORD"));
driver.get("https://api.ipify.org/");It rides on the Chrome DevTools Protocol, so it works with a local Chrome or Edge driver (a RemoteWebDriver needs wrapping with Augmenter first), and the selenium-devtools artifact on your classpath has to match the browser's major version. Firefox is not covered.
In Python, answer the challenge over BiDi, or generate a small extension. Selenium 4.32 added driver.network.add_auth_handler, which answers the proxy's 407 through WebDriver BiDi. It works on branded Chrome as well as Chrome for Testing and Edge, with one catch: navigate through BiDi too. With Selenium 4.52 and Chrome 156, a classic driver.get stalls at the proxy's challenge until it times out, while browsing_context.navigate loads the page:
options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
options.add_argument("--proxy-server=http://gw-rotating_datacenter.node4.io:8083")
options.enable_bidi = True
driver = webdriver.Chrome(options=options)
driver.network.add_auth_handler("USERNAME", "PASSWORD")
driver.browsing_context.navigate(
driver.current_window_handle, "https://api.ipify.org/", wait="complete")
print(driver.find_element("tag name", "body").text)Send later page loads through browsing_context.navigate as well, not driver.get. On Selenium older than 4.32 the route is a Manifest V3 extension, written to disk at runtime, that sets the proxy and answers onAuthRequired:
import json
import zipfile
from string import Template
from selenium import webdriver
WORKER = Template("""
chrome.proxy.settings.set({
value: {
mode: "fixed_servers",
rules: {
singleProxy: { scheme: "http", host: "$host", port: $port },
bypassList: ["localhost", "127.0.0.1"]
}
},
scope: "regular"
});
chrome.webRequest.onAuthRequired.addListener(
(details, callback) => {
callback({ authCredentials: { username: "$user", password: "$password" } });
},
{ urls: ["<all_urls>"] },
["asyncBlocking"]
);
""")
MANIFEST = {
"manifest_version": 3,
"name": "proxy auth",
"version": "1.0",
"permissions": ["proxy", "webRequest", "webRequestAuthProvider"],
"host_permissions": ["<all_urls>"],
"background": {"service_worker": "worker.js"},
}
def proxy_extension(host, port, user, password, path="proxy_auth.zip"):
with zipfile.ZipFile(path, "w") as z:
z.writestr("manifest.json", json.dumps(MANIFEST))
z.writestr("worker.js", WORKER.substitute(
host=host, port=port, user=user, password=password))
return path
options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
options.add_extension(proxy_extension(
"gw-rotating_datacenter.node4.io", 8083, "USERNAME", "PASSWORD"))
driver = webdriver.Chrome(options=options)
driver.get("https://api.ipify.org/")Four things break this in practice. The first is the browser. add_extension hands the archive to ChromeDriver, which unpacks it and starts the browser with the --load-extension switch, and branded Google Chrome dropped that switch in Chrome 137. On Chrome 137 or later the browser starts without the extension, so it uses no proxy at all and the IP echo shows your own address. Chrome for Testing, Chromium and Microsoft Edge still honor the switch, so the example above runs as written on them. On branded Chrome, install the same two files at runtime over WebDriver BiDi instead. That needs Selenium 4.34 or later and an unpacked folder, because Chrome's BiDi install refuses a zip:
import pathlib
def proxy_extension_dir(host, port, user, password, path="proxy_auth"):
folder = pathlib.Path(path)
folder.mkdir(exist_ok=True)
(folder / "manifest.json").write_text(json.dumps(MANIFEST))
(folder / "worker.js").write_text(WORKER.substitute(
host=host, port=port, user=user, password=password))
return str(folder.resolve())
options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
options.enable_bidi = True
options.enable_webextensions = True
driver = webdriver.Chrome(options=options)
driver.webextension.install(path=proxy_extension_dir(
"gw-rotating_datacenter.node4.io", 8083, "USERNAME", "PASSWORD"))
driver.get("https://api.ipify.org/")enable_webextensions adds the two launch flags a runtime install requires, --enable-unsafe-extension-debugging and --remote-debugging-pipe. The same approach also works on Chrome for Testing and Edge, so a suite that runs on more than one of them can use it everywhere. In both versions the proxy only exists once the extension is running, which is one more reason to end with the IP echo.
The old headless mode cannot load extensions at all. Before Chrome 132 that is what a bare --headless meant; since 132 a bare --headless is the new mode and the old one only survives as the separate chrome-headless-shell binary, which still cannot load them. --headless=new means the same thing on every version, which is why the examples use it. Manifest V2 examples, which are most of what a search turns up, no longer load in current Chrome; the webRequestAuthProvider permission is what makes the blocking auth listener legal under V3. And because the proxy is now set by the extension, do not also pass --proxy-server; two sources of proxy configuration are one more than you can debug.
Run a local forwarder that adds the header for you. selenium-wire was the standard answer for years: it starts a local proxy in front of the browser, injects Proxy-Authorization on the way out, and lets you swap the upstream mid-session with driver.proxy = {...}. It still works for many people, but the project was archived in January 2024 and receives no fixes for new Chrome or Python releases. It also intercepts TLS, which breaks against targets that pin certificates. Treat it as a compatibility tool for an existing suite, not the foundation of a new one. A credential-free local forwarder (any small proxy that accepts unauthenticated connections from localhost and authenticates upstream) does the same job without the TLS interception, at the cost of one more process to supervise.
If none of these lands and the browser still shows a 407, confirm the credentials in curl before touching Selenium again; fixing 407 Proxy Authentication Required walks that isolation step.
Three ways to rotate in Selenium
Selenium will not rotate for you, so the decision is where the rotation lives.
Restart the driver per proxy. The simplest mechanism, and for a job of a few hundred pages the right one. Keep a cycle over the addresses from your dashboard and launch a fresh driver per batch:
import itertools
from selenium import webdriver
PROXIES = ["203.0.113.10:8080", "203.0.113.11:8080", "203.0.113.12:8080"]
cycle = itertools.cycle(PROXIES)
def driver_for(proxy):
options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
options.add_argument(f"--proxy-server=http://{proxy}")
return webdriver.Chrome(options=options)
for batch in batches(urls, size=25):
driver = driver_for(next(cycle))
try:
for url in batch:
driver.get(url)
handle(driver)
finally:
driver.quit()The cost is a Chromium launch per rotation, which is hundreds of megabytes and a couple of seconds each time. The list above is static datacenter addresses with whitelisting on, so no auth code is needed; with credentials, generate the extension per proxy inside driver_for. The other cost is that the list is now yours to keep healthy: an address that times out stays in the cycle until you remove it.
Switch a local forwarder mid-session. With selenium-wire or your own forwarder in front of the browser, the upstream can change without restarting Chromium, so one long-lived driver walks a rotation. It is the only mechanism that gives you rotation and a persistent browser profile at once, which is useful when logging in is expensive. It inherits every caveat from the previous section.
Point the browser at a gateway and let the exit rotate behind it. One hostname, one launch, and the pool rotates for you: each new connection through gw-rotating_datacenter.node4.io (each HTTPS tunnel, or each plain-HTTP request) gets its own pick from the pool. No list to manage, no restarts, and pool health is the provider's problem. This is the mechanism for anything that runs unattended.
There is a wrinkle specific to browsers that the gateway does not cause but does expose. Rotation happens per connection, and a browser opens several of them to load one page, then keeps them alive for the next. A single page load through a rotating gateway can therefore fetch its document from one exit and its scripts from another, which is harmless on most sites and confusing on a site that binds a session to an address. The fix is to pin a session per driver: append a session segment to the username, and everything that driver loads shares one exit until you relaunch with a new id or the session reaches its ceiling: best effort up to 30 minutes on residential, and up to 30 minutes from first assignment on the rotating gateway.
import uuid
def session_driver(user, password):
session_id = uuid.uuid4().hex[:12]
options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
options.add_extension(proxy_extension(
"gw-residential.node4.io", 8082,
f"{user}-session-{session_id}", password))
return webdriver.Chrome(options=options)On branded Chrome 137 or later, replace the add_extension call with the BiDi route shown earlier: set enable_bidi and enable_webextensions on the options, then install proxy_extension_dir(...) with driver.webextension.install once the driver has started. Rotation is now a relaunch with a fresh id, which is the driver-restart pattern again, except the addresses come from the pool rather than a list you maintain. How long a pin holds and when you should let it go is in sticky versus rotating sessions.
Pinning is a first-class feature of the residential gateway. On the rotating gateways it depends on the gateway's session store: when that store is unreachable, a pinned request is refused with a 503 rather than served from a rotating exit. Test it before a job depends on it, with the check in the next section.
If the pin does not hold on a rotating gateway, read the X-Node4-Sticky header with curl -v through the gateway's HTTP port and the same username; none means no session segment was parsed. Selenium does not expose response headers, and for an HTTPS page the header is on the gateway's reply to the CONNECT, never inside the destination's HTTPS response; SOCKS5 carries no headers at all.
On the rotating gateways a typo in the session key is not always an error: straight after the username a misspelled key is a 407, and after a country segment the country code includes the misspelled option, so the request gets a 502. On residential a misspelled key is a 400 that names it. The targeting examples show each case.
| Mechanism | Rotates | Keeps browser state | Fits |
|---|---|---|---|
| Driver restart | Per launch | No | Small batches, static lists |
| Local forwarder | Any time | Yes | Long logged-in sessions |
| Gateway | Per connection, or per session id | Per driver | Unattended jobs, large crawls |
Verify the exit actually changed
Rotation that did not happen looks identical to rotation that did, right up until a target blocks the one address every worker has been sharing. Make the check part of the job:
def exit_ip(driver):
driver.get("https://api.ipify.org/")
return driver.find_element("tag name", "body").text.strip()
a = session_driver("USERNAME", "PASSWORD")
b = session_driver("USERNAME", "PASSWORD")
assert exit_ip(a) != exit_ip(b), "two sessions, one exit: pinning or rotation is off"Two drivers with different session ids should report different addresses; the same driver asked twice should report the same one. If the first assertion fails through a gateway, the session segment is being ignored or the plan does not include the gateway at all. If it fails with a proxy list, the list has duplicates or the flag did not take.
Where datacenter exits lose
A rotating datacenter pool solves per-address rate limits, and it does not solve classification. Targets that take bot traffic seriously look up the address's network and treat the whole hosting range as automation, so rotating within that range produces a different address with the same verdict. When a job that worked on a lenient site gets challenges or empty shells from a defended one, and the browser fingerprint is clean, that is usually what happened. The answer is a residential exit, whose address belongs to a consumer network, run with credentials so the session segment can hold it steady for the length of a flow.
Residential does not fix the other half. A headless browser announces itself through timing, missing plugins, the navigator.webdriver flag and a TLS handshake that does not match its claimed user agent, and a clean address under a flagged fingerprint still gets flagged. Tools like undetected-chromedriver address the fingerprint and do nothing for the proxy; treat the two as separate problems. Why your IP is getting blocked separates address reputation from fingerprint so you can tell which one you are looking at.
There is also the question of whether a browser is the right tool for the job at all. A browser fetches every script, image and font the page references, and through a metered plan that multiplier is on the bill. When the data is in the HTML or in a JSON endpoint, a Scrapy spider through the same gateway costs a fraction per page and rotates more cleanly; proxies for Scrapy covers that product choice. Save the browser for pages that genuinely need JavaScript to exist.
Leaks a browser adds
An HTTP client sends what you told it to. A browser resolves names, opens UDP sockets and prefetches, and only some of that follows the proxy setting.
- WebRTC discovers routes by sending UDP to STUN servers, outside the proxy, which lets a page's JavaScript learn your real address. Launch with
--force-webrtc-ip-handling-policy=disable_non_proxied_udpon Chromium, or setmedia.peerconnection.enabledto false in Firefox. - DNS through SOCKS5. With a
socks5://proxy Chromium hands the hostname to the proxy for the request itself, but prefetching and other lookups still run locally and reveal what you are about to visit; add--host-resolver-rules="MAP * ~NOTFOUND , EXCLUDE HOST"so every local lookup except the proxy's own name is refused and resolution happens at the exit. Firefox needsnetwork.proxy.socks_remote_dnsset to true. HTTP proxies do not have this problem, because the hostname travels inside the CONNECT request. - The bypass list. Anything on
--proxy-bypass-list(or Firefox'snetwork.proxy.no_proxies_on) resolves and connects directly. Keep it to localhost. - The profile. Timezone, language and cookies are the browser's own. An exit in one country and a profile claiming another is a signal on its own; align them per driver if the target cares.
Load a public WebRTC leak test page in the proxied browser after any change here. The only address on display should be the proxy exit.
The errors you will actually see
| Symptom | Cause | Fix |
|---|---|---|
ERR_PROXY_CONNECTION_FAILED | Wrong scheme, dead host, closed port | Test the same host and port with curl -x first |
ERR_TUNNEL_CONNECTION_FAILED | Proxy refused the CONNECT for an HTTPS target | Check credentials and whether the plan covers this endpoint |
| Auth dialog blocks the run | Credentials were in the flag | Whitelist, HasAuthentication, the BiDi auth handler, or the extension |
| Extension ignored | Branded Chrome 137 or later (no --load-extension), old headless (chrome-headless-shell, or bare --headless before Chrome 132), or a V2 manifest | webextension.install or Chrome for Testing, --headless=new, Manifest V3 |
| IP echo shows your own address | Flag did not take, or only Firefox http pair set | Re-check options; set the ssl pair |
| Same exit on every driver | Session id reused, or gateway not on the plan | Mint a new id per launch; check the plan |
Work the table top to bottom, and in every case start from curl. If curl -x http://HOST:PORT https://api.ipify.org/ fails, Selenium was never the problem, and proxy connection refused or timing out is the guide to read next.
Choosing the mechanism
For a test suite that needs a stable identity from a known address, whitelist a static datacenter proxy and pass the flag; there is nothing else to build. For a scraping job that runs unattended, point the browser at a gateway, pin a session per driver, and relaunch to rotate, escalating from the rotating gateway to residential on evidence of classification rather than in advance. Keep selenium-wire for the suite that already depends on it. Plans and what each includes are on the pricing page, and proxies for browser automation lays the same three products side by side for a Selenium, Puppeteer or Playwright job.