Python Proxy Integration: requests, httpx and aiohttp

Session pooling traps, the http:// scheme confusion, credential encoding, retries with backoff, and an egress check that proves traffic is proxied.

Python settled on three HTTP clients: requests for synchronous code, httpx when you want one API in both sync and async flavors, and aiohttp when your application already lives inside an event loop. All three route traffic through a proxy competently once configured, but each one is configured differently enough that assumptions carried from one to another are the main source of "the proxy doesn't work" reports we see from Python users.

This guide shows working configuration for current versions of each library, then covers the parts that cause real failures: the scheme confusion inside the requests proxy mapping, proxy settings applied after a session has already pooled connections, credentials that break URL parsing, timeouts, retries, environment-driven configuration, and the step people skip: proving that the egress address is ours and not yours. If your stack is Node rather than Python, the equivalent map is Node.js proxy setup.

Everything below assumes username and password authentication. If you whitelist your source IP instead, delete the credentials from every example and the rest holds unchanged.

The requests proxy mapping

requests takes proxy configuration as a dictionary mapping a URL scheme to a proxy address:

import requests

proxies = {
    "http":  "http://USERNAME:PASSWORD@HOST:PORT",
    "https": "http://USERNAME:PASSWORD@HOST:PORT",
}

r = requests.get("https://api.ipify.org", proxies=proxies, timeout=(10, 30))
print(r.text)

The keys name the scheme of the target URL, and the values name the proxy that traffic should traverse. Most people read the "https" line, reach instinctively for an https:// value, and break their setup in a way that produces no useful error message.

Your client speaks plain HTTP to the node4 endpoint. When the destination is an https:// URL, the library asks the proxy to open a raw tunnel with a CONNECT request, then performs the TLS handshake with the destination through that tunnel. The destination's certificate is verified by your machine exactly as it would be with no proxy in the path; the proxy relays encrypted bytes it cannot read. So the value under the "https" key still begins with http://: it addresses the proxy, and the proxy is a plain-HTTP listener. Write https:// there and the library attempts a TLS handshake against a socket that answers in plaintext, which surfaces as an SSLError or a wrapped ProxyError, and nothing in either message points at the one character responsible.

Sessions, pooling, and the ordering trap

Use a requests.Session for anything longer than a one-off script. Without one, every call opens a fresh TCP connection, and behind a proxy that is doubly expensive: each new connection pays for the hop to the proxy, a CONNECT round trip, and a TLS handshake before a single byte of your actual request moves.

The session's connection pool is also where a subtle ordering bug lives. Pooled sockets are keyed by how they were opened. A request made before you assigned proxy settings leaves behind an open, direct connection to that host, and later requests to the same host will happily reuse it, bypassing the proxy configuration you added in between. The symptom is confusing enough to be memorable: newly contacted hosts go through the proxy correctly, while the host you tested first keeps reporting your own address.

Two habits close the hole. Configure the session completely before it sends anything, and treat its proxy settings as fixed for its lifetime. If you need a different proxy mid-process (a different product, a different country), construct a new session and close the old one rather than mutating one that already holds sockets.

import requests

session = requests.Session()
session.proxies = {
    "http":  "http://USERNAME:PASSWORD@HOST:PORT",
    "https": "http://USERNAME:PASSWORD@HOST:PORT",
}

r = session.get("https://api.ipify.org", timeout=(10, 30))
print(r.text)

session.close()

A per-request proxies= argument also exists and overrides the session's setting for that single call. It is useful for the occasional direct request from an otherwise proxied session, and less useful as your primary mechanism, because it reintroduces the reuse problem in reverse.

Credentials that survive URL parsing

Because credentials ride inside the proxy URL, any reserved character in them changes what the URL means. The parser cannot distinguish an intentional separator from an unlucky password character; the parse simply comes apart, and the proxy rejects whatever fragments arrive. The full taxonomy of that failure (along with its cousin, the invisible whitespace a copy-paste drags in) is worked through in fixing 407 errors. Prevention is one function call:

from urllib.parse import quote

username = quote("USERNAME", safe="")
password = quote("p@ss:w/rd", safe="")

proxy = f"http://{username}:{password}@HOST:PORT"

Pass safe="" explicitly. The default for quote leaves / unescaped, because its usual job is encoding URL paths; inside a credential you want every reserved character converted. Encode each half on its own and then assemble the URL; running the assembled string through quote would also encode the separators that are supposed to remain separators.

httpx: one configuration for sync and async

httpx offers the same request API in both execution models, and proxy configuration belongs to the client constructor:

import httpx

proxy = "http://USERNAME:PASSWORD@HOST:PORT"
timeout = httpx.Timeout(30.0, connect=10.0)

with httpx.Client(proxy=proxy, timeout=timeout) as client:
    print(client.get("https://api.ipify.org").text)

The async client takes identical arguments:

import asyncio
import httpx

async def main() -> None:
    async with httpx.AsyncClient(proxy="http://USERNAME:PASSWORD@HOST:PORT") as client:
        r = await client.get("https://api.ipify.org")
        print(r.text)

asyncio.run(main())

Two version notes. Early httpx accepted a proxies= dictionary shaped like the requests one; current releases deprecate it in favor of a single proxy= URL, which matches how a node4 endpoint is used anyway. And when you genuinely need different routing per destination (one internal origin direct, everything else proxied), the mounts= argument maps URL patterns to transports:

client = httpx.Client(
    mounts={
        "all://": httpx.HTTPTransport(proxy="http://USERNAME:PASSWORD@HOST:PORT"),
        "all://internal.example.com": httpx.HTTPTransport(),
    }
)

Because the proxy is fixed at construction time, the ordering bug from the previous section cannot even be expressed in httpx: there is no attribute to assign after traffic has started. That is deliberate design, and a fair reason to pick httpx for new code.

aiohttp: the proxy is a request argument

aiohttp inverts the model. The ClientSession owns the connection pool, but the proxy is an argument to each request. Note that this argument is HTTP only: handing it a socks5:// URL raises ValueError, and SOCKS5 needs a different connector entirely, covered in SOCKS5 proxies with aiohttp.

import asyncio
import aiohttp

async def main() -> None:
    timeout = aiohttp.ClientTimeout(total=60, connect=10)
    auth = aiohttp.BasicAuth("USERNAME", "p@ss:w/rd")

    async with aiohttp.ClientSession(timeout=timeout) as session:
        async with session.get(
            "https://api.ipify.org",
            proxy="http://HOST:PORT",
            proxy_auth=auth,
        ) as resp:
            print(await resp.text())

asyncio.run(main())

Three aiohttp-specific points. First, prefer proxy_auth over embedding credentials in the proxy URL: BasicAuth takes the password verbatim, so the encoding section above stops applying at all. Second, the proxy URL must use the http:// scheme: aiohttp does not negotiate TLS with the proxy itself, which is the standard arrangement here anyway. Third, create one ClientSession per application and reuse it; a session per request throws away the pool that makes the library worth using, and does so at proxy prices.

The per-request model has a sharp edge in large codebases: forget the proxy argument on one call and that call silently goes direct. A small wrapper that always injects proxy and proxy_auth is cheap insurance against exactly the request you least wanted to leak your server's address.

Timeouts are not optional through a proxy

A proxied request has more places to stall than a direct one: the connection to the proxy, the tunnel establishment, the upstream connection, and the target's response. requests and aiohttp default to timeouts that are either infinite or generous enough to behave that way in a queue-processing loop; httpx, unusually and correctly, enforces a default. A worker thread or coroutine stuck on a dead request is the standard way a proxy integration degrades in production.

Set both phases explicitly. In requests the timeout is a (connect, read) tuple, and omitting it means wait forever. httpx has a structured httpx.Timeout with per-phase fields. aiohttp's ClientTimeout carries total and connect. Sensible values depend on your targets and how much delay your pipeline tolerates; the property that matters is that a hung request becomes a raised exception after a bounded wait, not a stalled worker you discover from a dashboard.

Retries, and why backoff earns its keep

Transient failures are part of the contract with any proxy pool: an exit leaving rotation mid-request, a target throttling an address, an upstream timeout. requests inherits a capable retry engine from urllib3; mount it once per session:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
session.proxies = {
    "http":  "http://USERNAME:PASSWORD@HOST:PORT",
    "https": "http://USERNAME:PASSWORD@HOST:PORT",
}

retry = Retry(
    total=3,
    backoff_factor=1.5,
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD"],
)

adapter = HTTPAdapter(max_retries=retry)
session.mount("http://", adapter)
session.mount("https://", adapter)

Two details earn their place in that snippet. allowed_methods restricts retries to idempotent verbs: replaying a POST because a hop timed out is how duplicate orders happen. And backoff_factor spaces the attempts out, which matters because a 429 answered instantly, with the same request from the same address, rarely goes better the second time.

On a rotating product the calculus improves: each fresh attempt through the gateway can leave from a different address, so a retry is not the same request from the same place but a genuinely new approach. When you want the opposite behavior (one exit held across a sequence of requests), that is a session-control question, covered in sticky vs rotating sessions; the products themselves are described on the rotating proxies page.

httpx retries connection failures if you ask (httpx.HTTPTransport(retries=3)) but deliberately does not retry on status codes; pair it with a library like tenacity when you need that. aiohttp ships no retry mechanism at all; the aiohttp-retry package or a hand-rolled loop around asyncio.sleep both work.

Reading configuration from the environment

All three libraries can pick up the standard proxy environment variables, with different levels of enthusiasm:

export HTTP_PROXY="http://USERNAME:PASSWORD@HOST:PORT"
export HTTPS_PROXY="http://USERNAME:PASSWORD@HOST:PORT"
export NO_PROXY="localhost,127.0.0.1,.internal"

requests and httpx read them by default: requests re-checks per request (disable with session.trust_env = False); httpx captures them when the client is constructed. aiohttp ignores them unless you opt in with ClientSession(trust_env=True), an omission that has surprised nearly everyone who has migrated an asyncio service. NO_PROXY carves out destinations that should stay direct, which you want for localhost health checks and internal services.

Environment configuration keeps credentials out of source control and lets identical code run proxied in production and direct in development. Its weakness is invisibility: nothing in the code says traffic is proxied, so a missing variable fails silently, and the traffic goes direct. Which is the argument for the final section.

Prove it: check the egress address

Every integration should end with the same test: ask an IP-echo service who you are, both ways.

import requests

PROXY = "http://USERNAME:PASSWORD@HOST:PORT"

direct = requests.get("https://api.ipify.org", timeout=10).text
proxied = requests.get(
    "https://api.ipify.org",
    proxies={"http": PROXY, "https": PROXY},
    timeout=(10, 30),
).text

print(f"direct:  {direct}")
print(f"proxied: {proxied}")
assert direct != proxied, "traffic is NOT going through the proxy"

Run it at deploy time, not just once during development. It catches the expired credential, the unset environment variable, and the session that pooled a direct connection, before any of them costs you scraped data attributed to your own server. On a rotating gateway, two proxied calls in a row may print different addresses; that is the product working, not a bug.

If the assertion fails, walk back through this guide in order: the scheme on the https key, proxy settings applied before first use, credentials encoded. If the proxied call errors with a 407 instead, that is well-mapped territory of its own; the 407 guide works the causes in order of likelihood.

Two scope notes to finish. Scrapy is deliberately absent from this page: its downloader-middleware layer wraps these same primitives in project-level machinery with its own conventions, and it has its own guide in Scrapy proxy middleware. And if your Python code drives a headless browser rather than an HTTP client, almost nothing here transfers: Chromium refuses credentials where every library above accepts them, and Puppeteer and Playwright proxies explains the mechanism that replaces them. For deciding which pool this code should point at in the first place (shared datacenter for volume, residential for targets that classify datacenter ranges), the pricing page lays the options side by side.

For a wider view, see our practical Requests guide.