Selenium Proxy Setup: Verify Routing and Handle Challenges

Route Chrome through a proxy from Selenium, read the exit inside the browser, wait for a real ready state and stop the job when a challenge appears.

This guide starts Chrome from Selenium (Python) through an HTTP proxy, reads the exit address from inside the browser, waits for an element that only appears when the task is ready, and stops the job when a known challenge element shows up. Every run ends with a verified result or a named failure: connection, unexpected exit, readiness timeout or challenge.

You need Python, Chrome, an HTTP proxy endpoint that accepts your machine without a browser login prompt (source-IP authorization, for example) and permission to automate the target. Handling a challenge here means detecting it and stopping. A proxy route does not stop a site from showing a CAPTCHA, and this guide does not solve or bypass one.

TL;DR

A selenium proxy setup controls browser routing, not whether a website shows a CAPTCHA.
Verify the browser exit address before running any authenticated task.
Keep proxy routing stable inside each browser session and rotate only between independent jobs.
Stop a challenged job instead of retrying it blindly.

Why this matters

A working proxy connection does not prove an automated workflow is accepted. Your browser can reach a site through the intended exit and still meet a challenge. Routing, authentication, page readiness and site access are four separate checks.

Test them separately. Otherwise a failed login becomes a proxy incident, a slow page becomes a timeout loop, and a CAPTCHA becomes an empty dataset.

The rotating proxies for web scraping guide covers where rotation belongs. Rotation should follow the boundaries of a task, not interrupt its logged-in state.

This guide uses Chrome, Python and an HTTP proxy endpoint that needs no browser-side login. Run it only against applications you own or have permission to automate. It does not cover fingerprint spoofing, paid CAPTCHA solving or defeating a site's restrictions.

Before you start

Prepare the runtime and targets. Install Python and Chrome. Have a permitted target page, a CSS selector that appears only when the page is ready, and an IP-check endpoint that returns JSON with an ip field. Use an endpoint you control or are allowed to call.
Prepare the proxy connection. Get the real hostname, port and access method. The browser example needs an HTTP endpoint that accepts your machine without a login prompt, for example one authorized by source IP.
Check the authentication gotcha first. Chrome's --proxy-server flag does not reliably accept a username and password embedded in the URL. If your endpoint needs credentials, arrange a supported method before starting Selenium. Never disable certificate validation to get around connection errors.

Keep secrets out of source control and screenshots. If you use source-IP authorization, confirm your runner has a stable outbound address. A laptop and a hosted worker rarely share one.

Set up your runtime and proxy configuration

Create an isolated environment and install Selenium:

python -m venv .venv
source .venv/bin/activate
python -m pip install selenium

On Windows, activate the environment with the command for your shell. Selenium Manager can fetch the matching driver when Chrome starts. Locked-down build environments need the browser and driver provisioned explicitly.

Supply the connection and test inputs as environment variables:

export PROXY_ENDPOINT='http://proxy-host:proxy-port'
export IP_CHECK_URL='https://your-authorized-ip-check.example/json'
export TARGET_URL='https://your-authorized-target.example/page'
export READY_SELECTOR='#your-ready-element'

Replace every value with your real configuration. READY_SELECTOR must match content that appears only when the required state is ready. A generic body selector proves HTML exists, not that your task succeeded.

Test the endpoint outside the browser:

curl --fail --show-error --silent --connect-timeout 10 --max-time 30 --proxy "$PROXY_ENDPOINT" "$IP_CHECK_URL"

The 10-second connect timeout and 30-second total timeout are example limits, not provider performance claims. Set them to fit your job. A failed connection ends setup here and does not trigger a batch of browser launches.

Record the returned exit address, the time, the Chrome version and the Selenium version actually installed. For a rotating endpoint, read the result against its documented session behavior instead of assuming the next connection reuses the same exit.

Expected result: curl reaches your IP-check endpoint through the proxy and prints a usable address. That proves command-line connectivity only. Selenium needs its own routing test.

Verify browser routing and the exit address

Save this script as browser_job.py. It starts a clean Chrome session, checks the browser's exit address, waits for your ready state, and closes the browser even when a step fails.

import ipaddress
import json
import os
from urllib.parse import urlsplit

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

proxy = os.environ["PROXY_ENDPOINT"]
ip_check_url = os.environ["IP_CHECK_URL"]
target_url = os.environ["TARGET_URL"]
ready_selector = os.environ["READY_SELECTOR"]
challenge_selector = os.getenv("CHALLENGE_SELECTOR")
expected_exit = os.getenv("EXPECTED_EXIT_IP")

endpoint = urlsplit(proxy)
if endpoint.scheme != "http" or not endpoint.hostname:
    raise ValueError("Use a valid HTTP proxy endpoint")
if endpoint.port is None:
    raise ValueError("Include the proxy port")
if endpoint.username is not None or endpoint.password is not None:
    raise ValueError("Configure authentication outside this example")

options = webdriver.ChromeOptions()
options.add_argument(f"--proxy-server={proxy}")

driver = webdriver.Chrome(options=options)
try:
    driver.set_page_load_timeout(30)
    driver.get(ip_check_url)
    body = driver.find_element(By.TAG_NAME, "body").text
    exit_ip = str(ipaddress.ip_address(json.loads(body)["ip"]))
    if expected_exit:
        expected = str(ipaddress.ip_address(expected_exit))
        if exit_ip != expected:
            raise RuntimeError("Unexpected proxy exit address")
    print(json.dumps({"event": "browser_exit", "ip": exit_ip}))

    driver.get(target_url)

    def application_ready(browser):
        if challenge_selector:
            challenges = browser.find_elements(
                By.CSS_SELECTOR, challenge_selector
            )
            if any(el.is_displayed() for el in challenges):
                raise RuntimeError("Challenge detected; stop this job")
        elements = browser.find_elements(By.CSS_SELECTOR, ready_selector)
        return any(el.is_displayed() for el in elements)

    WebDriverWait(driver, 20).until(application_ready)
    print(json.dumps({"event": "application_ready"}))
finally:
    driver.quit()

Run python browser_job.py. Read the browser_exit event before you trust the page result. An address shown in a provider dashboard does not prove this browser used it.

Set EXPECTED_EXIT_IP only when the workflow needs a known exit and your connection arrangement supports one. Leave it unset when validating a rotating endpoint.

Expected result: the browser prints a valid exit address, reaches the authorized target and confirms the ready condition. A connection failure, malformed probe response, unexpected exit or readiness timeout fails the job.

The script uses a 30-second page-load timeout and a 20-second readiness wait. They are separate controls. A page can finish loading before the content you need renders, and no wait rescues a failed proxy connection.

Handle sessions and challenges

Define the job boundary before adding concurrency. One browser job is one complete permitted transaction. Keep its proxy route unchanged while it runs.

Set CHALLENGE_SELECTOR when your own application exposes a known challenge container. The script checks its visibility during the readiness wait. It is not a universal CAPTCHA detector for arbitrary sites.

Treat a detected challenge as a stop condition. Record the task, close the browser and route it to whatever handling process you agreed with the site owner. Refreshing a challenged page repeatedly proves nothing.

Log failure classes separately: connection errors, proxy authentication failures, readiness timeouts, access denials and challenges. Each needs a different fix, and lumping them hides the cause.

Save sanitized diagnostics on failure: current URL, browser version, exception class and task ID. Screenshots and page source can hold personal data, so store them under your data-handling policy.

Expected result: every task ends with a verified result or an explicit failure reason. A challenge page never becomes a successful empty export.

Cookies live in the browser. Exit selection happens at the proxy. Changing one does not reset the other. For stateful work, write both lifetimes into your runbook and close the browser when the intended session ends.

Variant: rotate between independent jobs

Rotate between jobs when each task is independent and the permitted workflow needs no continuing identity. Get a newly assigned session using your provider's documented mechanism, launch a fresh browser and repeat exit verification.

Do not invent session parameters or append guessed suffixes to a proxy username. Formats differ by provider. The code above consumes a configured endpoint and does not create sticky sessions.

Routing pattern	Best for	Advantage	Limitation
Stable exit during a browser job	Authorized login flows, multi-step tasks	Routing matches browser state	Needs an exit or session that lasts the whole task
Rotation between browser jobs	Independent authorized collection	Routing changes never hit live browser state	Challenges can still occur; needs fresh verification each time

Pick the pattern from the task's state requirements. A proxy changes the network route; it does not change whether the target classifies the session as automated, and it does not make an unauthorized request authorized.

Give each worker its own driver instance. Never share one live WebDriver across unrelated tasks, and never change the proxy route under an active login. If you plan to use both patterns, run an acceptance test for each, including cleanup after failures. An orphaned browser keeps eating memory after the queue says the job is done.

For datacenter-based routing options, see the datacenter proxy provider comparison. For exits that change per connection, see the rotating proxy API roundup.

Troubleshooting

The browser opens a proxy authentication prompt

The endpoint requires credentials this example does not supply. Switch to a supported access method such as source-IP authorization, or run an approved local forwarding proxy that handles upstream login. Keep any forwarding listener bound to the machine that needs it.

A successful authenticated curl test does not prove Chrome received those credentials. Retest in the browser after every change.

The IP-check response is not valid JSON

Inspect what your probe returns. The script expects a JSON object with an ip field. An HTML error page, login redirect or plain-text address breaks that contract.

Fix the probe or adapt the parser to its documented format. Never replace a parse error with an assumed address.

The page loads but the readiness wait times out

Check READY_SELECTOR against the rendered page. A renamed element, an iframe or content inside a shadow root needs a different locator strategy.

Do not raise the timeout until you know the selector can match. A longer wait does not fix a selector that never will.

The target still shows a CAPTCHA

Stop the job and review the authorized workflow with the site owner. A correct proxy route does not guarantee access, and cycling exits does not diagnose an application policy.

In your own test environment, use the challenge vendor's documented test configuration, and keep it separate from production.

Customize your workflow

Build the next layer around completed tasks, not browser launches. Log task ID, exit verification result, readiness result and cleanup result. Keep credentials and sensitive page content out of routine logs.

Start with one worker. Add concurrency only within your authorization and connection limits, and confirm each worker finishes and closes cleanly before adding the next.

Version your selectors next to the script. A site's markup changes independently of your proxy configuration, and your deployment checks should catch a broken ready condition before a scheduled run exports misleading data.

Node4's proxies for Puppeteer, Playwright and Selenium carry the whitelist and session-id targeting this workflow assumes, on rotating datacenter, Rotating Unmetered or residential exits; see pricing for what each plan includes.

If the target offers an approved API or export, evaluate it before expanding browser automation. Use Selenium when the browser interaction is actually required. For headless frameworks beyond Selenium, read the Puppeteer and Playwright proxy guide.

FAQ

What is the correct Selenium proxy setup for Chrome?

Pass a proxy endpoint to ChromeOptions with the --proxy-server argument, then verify the exit address from inside the browser. The example uses an HTTP endpoint without browser-side login and fails the job if verification fails.

Does a Selenium proxy prevent CAPTCHAs?

No. A proxy controls routing and does not guarantee CAPTCHA-free access. Stop challenged jobs and use the site's approved access or testing process.

Can I put a username and password in Chrome's proxy-server setting?

Do not rely on embedded credentials there. Configure a supported authentication method separately, such as source-IP authorization, and test it in the browser before running tasks.

Should I rotate the proxy after every Selenium request?

No, keep routing stable during a stateful browser job. Rotate between independent jobs when that fits the permitted workflow, then verify the new browser's exit.

Does headless Chrome reduce CAPTCHAs?

Not by itself. Headless mode changes how Chrome runs, not whether a site accepts automation. Validate routing and application behavior in the same mode your production job uses.

How do I detect a CAPTCHA in Selenium without solving it?

For an application you control, detect a known visible challenge element and stop the job. No universal CSS selector identifies every site's CAPTCHA.

What should I retest after updating Selenium or Chrome?

Retest proxy access, browser exit verification, readiness, challenge handling and browser cleanup. Compare against the Chrome and Selenium versions you recorded when the setup last worked.

One last thing

An empty result is not evidence that a page has no data. It can mean your browser hit a login screen, a challenge or the wrong application state. Require a positive readiness condition before extraction, then validate the extracted fields before marking a task complete.

That last check separates a browser that ran from a workflow that worked.