Python Proxy Integration: requests, httpx and aiohttp

Working proxy configuration for requests, httpx and aiohttp, then the details that cause failures: the http:// scheme, session pooling, encoding, timeouts.

Every Python client takes the same proxy URL, http://USERNAME:PASSWORD@HOST:PORT, copied from the Connect card on the dashboard's Proxies page, and it starts with http:// even when the target is https://. Find your client below, then read the details that cause real failures: proxy settings applied after a session has pooled connections, credentials that break URL parsing, timeouts, retries, environment variables, and the check that proves traffic is leaving through the proxy. If you whitelist your source IP instead of using credentials, drop USERNAME:PASSWORD@ and the rest holds. If your stack is Node rather than Python, see Node.js proxy setup.

requests

import requests

proxies = {
    "http":  "http://USERNAME:PASSWORD@HOST:PORT",
    "https": "http://USERNAME:PASSWORD@HOST:PORT",
}

r = requests.get("https://api.ipify.org", proxies=proxies, timeout=(10, 30))
print(r.text)

The keys name the scheme of the target URL; the values name the proxy. The value under "https" still begins with http://, because your client speaks plain HTTP to the Node4 endpoint: for an https:// destination the library asks the proxy for a CONNECT tunnel and performs the TLS handshake with the destination through it, so the destination's certificate is verified by your machine as usual and the proxy relays encrypted bytes it cannot read. Write https:// there and the library attempts a TLS handshake against a socket that answers in plaintext, which surfaces as an SSLError or a wrapped ProxyError with nothing in the message pointing at the scheme.

For anything longer than a one-off script, use a Session, which reuses connections. Behind a proxy that matters twice over: each new connection pays for the hop to the proxy, a CONNECT round trip and a TLS handshake before your request moves.

import requests

session = requests.Session()
session.proxies = {
    "http":  "http://USERNAME:PASSWORD@HOST:PORT",
    "https": "http://USERNAME:PASSWORD@HOST:PORT",
}

r = session.get("https://api.ipify.org", timeout=(10, 30))
print(r.text)

session.close()

Configure the session before it sends anything. Pooled sockets are keyed by how they were opened. A request made before you assign proxy settings leaves an open direct connection to that host, and later requests to the same host reuse it, bypassing the proxy you added in between. The symptom: newly contacted hosts go through the proxy, while the host you tested first keeps reporting your own address. Treat a session's proxy settings as fixed for its lifetime; if you need a different proxy mid-process, create a new session and close the old one. A per-request proxies= argument overrides the session's setting for one call, which is useful for an occasional direct request and less useful as the primary mechanism.

httpx

The same request API in sync and async form, with the proxy fixed in the client constructor:

import httpx

proxy = "http://USERNAME:PASSWORD@HOST:PORT"
timeout = httpx.Timeout(30.0, connect=10.0)

with httpx.Client(proxy=proxy, timeout=timeout) as client:
    print(client.get("https://api.ipify.org").text)

import asyncio
import httpx

async def main() -> None:
    async with httpx.AsyncClient(proxy="http://USERNAME:PASSWORD@HOST:PORT") as client:
        r = await client.get("https://api.ipify.org")
        print(r.text)

asyncio.run(main())

Two version notes. Early httpx accepted a proxies= dictionary shaped like the requests one; current releases deprecate it in favor of a single proxy= URL. For different routing per destination (one internal origin direct, everything else proxied), mounts= maps URL patterns to transports:

client = httpx.Client(
    mounts={
        "all://": httpx.HTTPTransport(proxy="http://USERNAME:PASSWORD@HOST:PORT"),
        "all://internal.example.com": httpx.HTTPTransport(),
    }
)

Because the proxy is fixed at construction, the session-ordering problem above cannot occur in httpx.

aiohttp

The ClientSession owns the connection pool, but the proxy is an argument to each request. That argument is HTTP only: a socks5:// URL raises ValueError, and SOCKS5 needs a different connector, covered in SOCKS5 proxies with aiohttp.

import asyncio
import aiohttp

async def main() -> None:
    timeout = aiohttp.ClientTimeout(total=60, connect=10)
    auth = aiohttp.BasicAuth("USERNAME", "p@ss:w/rd")

    async with aiohttp.ClientSession(timeout=timeout) as session:
        async with session.get(
            "https://api.ipify.org",
            proxy="http://HOST:PORT",
            proxy_auth=auth,
        ) as resp:
            print(await resp.text())

asyncio.run(main())

Three points. Prefer proxy_auth over embedding credentials in the URL: BasicAuth takes the password verbatim, so the encoding step in the next section does not apply. The proxy URL must use http://, since aiohttp does not negotiate TLS with the proxy. And create one ClientSession per application and reuse it; a session per request throws away the pool. Because the proxy is per request, a call that omits it goes direct silently; a small wrapper that always injects proxy and proxy_auth is cheap insurance.

Credentials that survive URL parsing

Credentials inside the proxy URL are parsed as URL parts, so a reserved character in a password changes what the URL means and the proxy rejects the fragments that arrive. Fixing 407 errors covers that failure and the invisible whitespace a copy-paste drags in. Prevention is one function call:

from urllib.parse import quote

username = quote("USERNAME", safe="")
password = quote("p@ss:w/rd", safe="")

proxy = f"http://{username}:{password}@HOST:PORT"

Pass safe="" explicitly; the default leaves / unescaped because quote usually encodes paths. Encode each half separately and then assemble the URL, so the separators stay separators.

Timeouts

A proxied request has more places to stall than a direct one: the connection to the proxy, the tunnel, the upstream connection and the target's response. requests and aiohttp default to timeouts that are infinite or close to it; httpx enforces a default. Set both phases explicitly: in requests the timeout is a (connect, read) tuple, httpx has a structured httpx.Timeout, and aiohttp's ClientTimeout carries total and connect. Values depend on your targets; what matters is that a hung request becomes an exception after a bounded wait rather than a stalled worker.

Retries and backoff

Transient failures are part of using any proxy pool: an exit leaving rotation mid-request, a target throttling an address, an upstream timeout. requests inherits a retry engine from urllib3; mount it once per session:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
session.proxies = {
    "http":  "http://USERNAME:PASSWORD@HOST:PORT",
    "https": "http://USERNAME:PASSWORD@HOST:PORT",
}

retry = Retry(
    total=3,
    backoff_factor=1.5,
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD"],
)

adapter = HTTPAdapter(max_retries=retry)
session.mount("http://", adapter)
session.mount("https://", adapter)

allowed_methods restricts retries to idempotent verbs, so a POST is not replayed because a hop timed out. backoff_factor spaces the attempts, which matters because a 429 retried instantly from the same address rarely goes better.

On a rotating gateway the exit belongs to the connection. A plain-HTTP request gets a new exit every time, but an HTTPS request travels inside a tunnel, and a Session keeps that tunnel open: a status-code retry goes out through the same tunnel and the same exit. If a retry should come from a different address, send it on a new connection, for example from a fresh Session, rather than relying on the retry adapter. When you want the opposite (one exit held across a sequence), that is a session question, covered in sticky vs rotating sessions; the products are on the rotating proxies page.

httpx retries connection failures if asked (httpx.HTTPTransport(retries=3)) but not on status codes; pair it with tenacity for that. aiohttp ships no retry mechanism; aiohttp-retry or a loop around asyncio.sleep both work.

Environment variables

All three libraries can read the standard proxy variables:

export HTTP_PROXY="http://USERNAME:PASSWORD@HOST:PORT"
export HTTPS_PROXY="http://USERNAME:PASSWORD@HOST:PORT"
export NO_PROXY="localhost,127.0.0.1,.internal"

requests and httpx read them by default: requests re-checks per request (disable with session.trust_env = False); httpx captures them when the client is constructed. aiohttp ignores them unless you opt in with ClientSession(trust_env=True). NO_PROXY keeps listed destinations direct, which you want for localhost health checks and internal services.

Environment configuration keeps credentials out of source control and lets the same code run proxied in production and direct in development. Its weakness is that a missing variable fails silently and traffic goes direct, which is why the final check matters.

Check the egress address

End every integration with the same test: ask an IP-echo service who you are, both ways.

import requests

PROXY = "http://USERNAME:PASSWORD@HOST:PORT"

direct = requests.get("https://api.ipify.org", timeout=10).text
proxied = requests.get(
    "https://api.ipify.org",
    proxies={"http": PROXY, "https": PROXY},
    timeout=(10, 30),
).text

print(f"direct:  {direct}")
print(f"proxied: {proxied}")
assert direct != proxied, "traffic is NOT going through the proxy"

Run it at deploy time, not only during development. It catches an expired credential, an unset environment variable and a session that pooled a direct connection. On a rotating gateway, two proxied calls on separate connections may print different addresses; that is the product working.

If the assertion fails, walk back through this guide: the scheme on the https key, proxy settings applied before first use, credentials encoded. If the proxied call returns a 407 instead, the 407 guide works the causes in order of likelihood.

Scrapy is not covered here: its downloader middleware wraps these primitives in project-level machinery with its own conventions, in Scrapy proxy middleware. If your Python code drives a headless browser rather than an HTTP client, Chromium refuses credentials where every library above accepts them; Puppeteer and Playwright proxies explains the mechanism that replaces them. For choosing which product this code should point at (shared datacenter for volume, residential for targets that classify datacenter ranges), the pricing page lays the options side by side, and the practical Requests guide gives a wider view.

Runnable versions of these patterns (requests, httpx, aiohttp, SOCKS5, sticky sessions, retries and a static proxy checker) are in the Python folder of our examples repository on GitHub.