Python Requests Proxy Sessions: Timeouts, Retries and Resume

Configure a python requests proxy with persistent cookies, explicit timeouts and checkpoints. Keep scraping sessions consistent and recoverable.

This guide builds a requests.Session that routes through your proxy with explicit connect and read timeouts, at most three attempts for a safe GET, and a checkpoint file that lets a stopped job resume at the next unfinished page. The outcome is recoverability: when a request fails or the process dies, the rerun picks up where the last committed page ended and does not duplicate records.

You need Python, the Requests library, the proxy host, port and credentials, and permission to collect from the target. Interruptions still happen. A proxy does not keep a target serving you, and a Requests session does not hold a rotating exit in place, so the design assumes failures and makes restarting cheap.

TL;DR

Use requests.Session for a python requests proxy workflow with persistent cookies and connection reuse.
Match proxy rotation to task boundaries, not individual requests in an authenticated session.
Set explicit timeouts, retry safe requests selectively and save progress after successful processing.
Keep TLS verification enabled and keep proxy credentials out of logs.

Why this matters

A scraping session has separate layers: your Python connection pool, the website's cookies and the proxy's exit assignment. Preserving one does not preserve the others. A cookie jar can survive while the exit IP changes.

Node4 fits developers who need proxies and manage Python scraping sessions themselves. You still implement request handling, checkpoints and application-specific validation. A proxy transports requests; it does not execute JavaScript or decide whether a response contains usable records.

For a stateful task, keep the exit consistent while the task runs. For independent public pages, rotation belongs between tasks. The rotating proxies for web scraping guide explains that separate workflow.

In a scraping pipeline, success means more than receiving a response. Validate the expected content, save it safely and record completed work before moving on.

Before you start

Have permission to collect the target data, an authorized target URL and the proxy host, port, protocol and authentication details. Use credentials supplied for your endpoint; do not invent session parameters.
Create an isolated Python environment and install Requests. Install its SOCKS extra only if you use a SOCKS endpoint. Keep credentials outside source control, shared notebooks and debug output.
Confirm how long the exit assignment lasts. A Requests session does not pin a rotating proxy's exit IP. Sticky assignment and cookie persistence are separate controls, and restarting Python does not automatically restore either.

Use the following commands to prepare the environment:

python -m venv .venv
source .venv/bin/activate
python -m pip install requests

The activation command is for a POSIX shell. On Windows PowerShell, activate the same environment with .venv\Scripts\Activate.ps1.

Start with one worker and one authorized URL. Validate the route before adding pagination or scheduling. Otherwise, a configuration error becomes repeated failed traffic.

Proxy endpoint configuration

Supply credentials without hard-coding them

Set TARGET_URL to the authorized page you want to fetch.
Set PROXY_HOST and PROXY_PORT to the endpoint supplied by your provider. The host value must contain only the hostname, not a scheme or path.
Set PROXY_USER and PROXY_PASSWORD if the endpoint uses username/password authentication.
Set PROXY_SCHEME to match the endpoint protocol. Do not choose a protocol from the target website's URL.

This Bash setup prompts for your values instead of putting a password directly into a command saved in shell history:

read -r -p 'Authorized target URL: ' TARGET_URL
read -r -p 'Proxy hostname: ' PROXY_HOST
read -r -p 'Proxy port: ' PROXY_PORT
read -r -p 'Proxy username: ' PROXY_USER
read -r -s -p 'Proxy password: ' PROXY_PASSWORD
printf '\n'
read -r -p 'Proxy scheme: ' PROXY_SCHEME
export TARGET_URL PROXY_HOST PROXY_PORT
export PROXY_USER PROXY_PASSWORD PROXY_SCHEME

Environment variables avoid committing credentials into a script. They are not a secret vault. Use your deployment platform's secret storage for scheduled jobs, and avoid printing the process environment.

Choose the correct protocol

Requests uses the dictionary key to identify the destination scheme. The value describes the connection to the proxy. An HTTPS destination can therefore use an http:// proxy URL, with HTTPS tunneled through the HTTP proxy.

Proxy URL scheme	Best for	What it does	Limitation
`http://`	A supplied HTTP proxy endpoint	Uses HTTP proxying, including CONNECT for HTTPS destinations	Does not itself encrypt the client-to-proxy connection
`socks5://`	SOCKS workflows requiring local DNS	Resolves the destination hostname locally	Destination DNS resolution happens outside the proxy
`socks5h://`	SOCKS workflows requiring proxy-side DNS	Sends the hostname to the SOCKS proxy for resolution	Requires SOCKS support and a compatible endpoint

For either SOCKS scheme, install the dependency:

python -m pip install 'requests[socks]'

Expected result: you have one known endpoint, a matching protocol and credentials available to the process. Do not proceed until those values match the provider's configuration.

Requests session configuration

Create the reusable client

Save the following code as proxy_client.py.
Build one session per sequential worker or logical task.
Set trust_env to False so environment proxy settings do not silently replace your explicit route.
Keep certificate verification enabled. For a private certificate authority, configure its trusted CA bundle rather than disabling verification.

import os
from urllib.parse import quote

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry


def build_session():
    scheme = os.environ['PROXY_SCHEME']
    if scheme not in {'http', 'https', 'socks5', 'socks5h'}:
        raise ValueError('Unsupported proxy scheme')

    host = os.environ['PROXY_HOST']
    port = int(os.environ['PROXY_PORT'])
    if not 1 <= port <= 65535:
        raise ValueError('Invalid proxy port')

    user = os.environ.get('PROXY_USER', '')
    password = os.environ.get('PROXY_PASSWORD', '')
    auth = ''
    if user or password:
        auth = f'{quote(user, safe="")}:{quote(password, safe="")}@'
    proxy_url = f'{scheme}://{auth}{host}:{port}'

    session = requests.Session()
    session.trust_env = False
    session.proxies.update({
        'http': proxy_url,
        'https': proxy_url,
    })

    retry = Retry(
        total=2,
        connect=2,
        read=2,
        status=2,
        allowed_methods=frozenset({'GET'}),
        status_forcelist=(502, 503, 504),
        backoff_factor=0.5,
        respect_retry_after_header=False,
        raise_on_status=False,
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)
    return session

Credential encoding matters. An @ or : inside a raw password can change how a proxy URL is parsed. quote(..., safe='') encodes those characters before constructing the URL.

Setting trust_env=False also disables environment-derived configuration such as automatic .netrc authentication. Configure any required target authentication explicitly. Never reuse proxy credentials as website credentials.

Verify one request before scraping

Run this check against the authorized URL you supplied:

python - <<'PY'
import os
from proxy_client import build_session

with build_session() as session:
    response = session.get(
        os.environ['TARGET_URL'],
        timeout=(5, 30),
        allow_redirects=False,
    )
    print('HTTP status:', response.status_code)
    print('Content type:', response.headers.get('Content-Type'))
    response.raise_for_status()
PY

The example selects a 5-second connect timeout and a 30-second read timeout. These are starting settings, not measured performance claims. The read timeout limits waiting between received data, not the total duration of a job.

Redirects are disabled for this first check. Inspect any redirect before enabling it, especially if it changes the hostname or takes you to a login page. A redirect response does not prove that your target content was fetched.

Expected result: the client reaches the intended destination through the configured endpoint. Confirm the egress route with a diagnostic endpoint you control or one supplied by your provider before relying on location-sensitive results.

Retry and checkpoint configuration

Separate transport failures from application failures

Retry safe GET requests only within a bounded policy.
Stop on authentication failures, access denials and rate limits instead of repeatedly changing exits.
Validate the response body against the target's documented format.
Save a checkpoint only after processing and storing the records successfully.

The adapter permits 2 retries, meaning at most 3 attempts for a retryable request. It selects gateway-related responses 502, 503 and 504, alongside retryable connection and read failures.

This example deliberately does not retry 429 responses automatically. It also disables automatic Retry-After handling in the adapter. A production scheduler should inspect a rate-limit response, honor the target's instructions and defer the task rather than immediately replay it.

An HTTP 200 response is not a scraping success criterion. A login page or challenge page can arrive with that status. Check required fields, expected page markers or a documented JSON schema before marking work complete.

Save progress at the processing boundary

The following pattern records the next page after your own processing function succeeds. Use it only when the target documents a page parameter. Other targets require cursors, offsets or explicit pagination links.

import json
import os
from pathlib import Path

from proxy_client import build_session


def run_pages(process_page, page_limit):
    checkpoint = Path('progress.json')
    if checkpoint.exists():
        page = json.loads(checkpoint.read_text())['next_page']
    else:
        page = 1

    with build_session() as session:
        while page <= page_limit:
            response = session.get(
                os.environ['TARGET_URL'],
                params={'page': page},
                timeout=(5, 30),
            )
            response.raise_for_status()
            process_page(response)

            temporary = checkpoint.with_suffix('.tmp')
            temporary.write_text(json.dumps({'next_page': page + 1}))
            temporary.replace(checkpoint)
            page += 1

Implement process_page to reject unexpected content and commit extracted records. Pass a page limit established from the target's pagination, not a guessed stopping point. This is an integration pattern, not a universal parser.

The temporary file and replacement reduce the risk of leaving a partially written checkpoint. They do not create a transaction with your database. If the process stops after saving records but before saving progress, the next run fetches that page again.

Use stable record identifiers and an idempotent write operation to handle that replay. For authenticated targets, reestablish authorized authentication after a restart: the checkpoint stores progress, not cookies.

Expected result: a failed request leaves the current page unfinished. Restarting resumes from the saved position, while duplicate-safe storage protects against replay at the commit boundary.

Independent-page workflow

Rotate between tasks, not inside a task

Use a separate workflow for independent pages that do not depend on a shared login, cart or pagination session. Complete one task, save its result and then start the next task with the appropriate exit assignment.

Creating a new requests.Session resets its client-side cookie jar and connection pool. It does not instruct a proxy service to choose another exit. Use the provider's documented rotation controls; never add an invented session token to the username.

A stable exit supports continuity but keeps failures tied to that exit. Rotation separates independent tasks but can disrupt IP-bound application state. Choose according to the target's documented behavior, not an assumption that more rotation improves success.

For a scheduled collector, use a task queue with completion state rather than one infinite request loop. Keep each worker's session separate. Increase concurrency only after measuring response validity, rate-limit responses and timeouts under permitted traffic levels.

Troubleshooting

Proxy authentication fails

An HTTP 407 indicates that the proxy requires authentication. Confirm the credentials, permitted authentication method and endpoint. For IP-authorized access, verify that the worker's outbound IP is authorized; do not assume your laptop and deployment server share one address.

The session loses continuity

Check whether the exit changed while cookies remained intact. Sticky assignments expire independently of Python sessions. Split long tasks into safe boundaries or use an endpoint whose documented behavior fits the task; sending another request does not necessarily extend an assignment.

Requests raises a certificate error

Check the target certificate, local trust store and any authorized inspection infrastructure. Supply a trusted CA bundle when required. Do not fix this with verify=False: that removes certificate verification rather than correcting the trust problem.

Requests times out repeatedly

Test one request without concurrency. Separate connection failures from slow response reads, and compare the result with an authorized direct request where appropriate. Raising the timeout hides neither routing problems nor a target that refuses the workload.

Responses succeed but records disappear

Inspect the body before parsing. Confirm its content type, expected structure and authentication state. Reject unexpected pages explicitly, and avoid logging complete response bodies when they contain personal data, tokens or session cookies.

Customize your workflow

Keep routing, authentication, parsing and persistence as separate components. That separation lets you replace a proxy endpoint without rewriting extraction logic and change storage without changing request handling.

Add structured logs containing task identifiers, status codes, attempt outcomes and processing results. Exclude proxy passwords, authorization headers and cookies. Measure valid records written, not just requests completed.

For persistent deployments, move the file checkpoint into your task store and make completion updates transactional where possible. Set a job-level deadline separately from Requests' connect and read timeouts.

Node4's residential proxies take a session id in the username. Residential sessions are best effort, with a 30-minute ceiling, and a route that keeps failing needs a new session id; the checkpoint above is what lets a resumed job switch sessions without refetching finished pages. See pricing for the residential plans.

FAQ

How do I configure a python requests proxy?

Set the session's proxies mapping with the proxy URL under both http and https destination keys. Use the endpoint's supplied protocol and authentication details, then test one authorized request before collecting data.

Does requests.Session keep the same proxy IP?

No. requests.Session preserves cookies and reuses connections, but the proxy service controls exit assignment. Use documented sticky or static endpoint behavior when your task requires a consistent exit.

Can I use an HTTP proxy for an HTTPS website?

Yes. An HTTP proxy can tunnel an HTTPS destination through CONNECT. Keep TLS verification enabled and distinguish the target URL scheme from the proxy connection scheme.

What's the difference between socks5 and socks5h in Requests?

The socks5 scheme resolves destination hostnames locally, while socks5h asks the proxy to resolve them. Both require the Requests SOCKS dependency and a compatible endpoint.

Should I retry every failed scraping request?

No. Retry selected transient failures for safe requests, with a fixed limit. Authentication errors, access denials and rate limits require corrective action or deferred scheduling, not repeated immediate requests.

How do I resume a scraper after it stops?

Save the next unfinished task after successfully committing its records. Make record writes duplicate-safe, and reestablish any required authentication because a progress checkpoint does not restore session cookies.

Can a proxy make Requests render JavaScript?

No. A proxy routes traffic; Requests does not execute page JavaScript. Use a documented data endpoint or an authorized browser workflow when the required content depends on browser execution.

One last thing

A retry repeats a request, not necessarily the same website state. If your cookies expire or the exit changes, replaying a URL can return different content even when the transport succeeds.

Before scheduling the collector, stop it deliberately after a successful write and restart it. Verify that completed records remain intact, replay produces no duplicates and authentication is restored correctly. Recovery is a behavior you test, not a property you obtain by enabling retries.