Go net/http Proxy: Authenticated Transport and Timeouts
Configure a golang http proxy with encoded credentials and a reusable transport, then verify the exit address and diagnose 407 errors.
Instead of manually adding proxy credentials to every request, configure a reusable Go http.Transport with an authenticated proxy URL and attach it to one http.Client. This workflow gives you a runnable golang http proxy client with bounded timeouts, response-size checks, and explicit handling for failed requests.
TL;DR
- Configure golang http proxy authentication with url.UserPassword and http.ProxyURL, not an origin Authorization header.
- Reuse one transport, bound request duration, and close every response body.
- Verify the proxy exit separately from the target website's response.
Proxy authentication and website authentication are different connections
Proxy authentication and website authentication belong to different connections. Mixing them sends credentials to the wrong place or produces an authentication failure before your scraper reaches the target.
A working proxy also does not prove a working scrape. Separate connection failures, proxy authentication failures, target HTTP errors, and unexpected page content. That distinction tells you what to fix instead of repeatedly changing credentials.
Node4's shared datacenter and other HTTP/SOCKS5 proxies for scraping and data collection are on Node4's shared proxies page.
Start with the curl authenticated proxy workflow if you need to isolate the endpoint before debugging Go. Then use the same endpoint and credentials in the program below.
Before you start
- Install Go and use a Bash-compatible terminal for the commands. Have a proxy endpoint, username, and password, plus an HTTPS URL you are authorized to request.
- Confirm the proxy endpoint's actual scheme and port. An HTTP proxy can carry requests to an HTTPS target through CONNECT; the target's scheme does not determine the proxy's scheme.
- Check credential encoding before testing. Passwords containing
@,:, or/must not be concatenated into a URL manually. The program usesurl.UserPasswordto encode credentials correctly.
Use a controlled HTTPS endpoint for the first request. Prefer an endpoint you operate that returns the connecting client's IP address. A successful request alone proves connectivity, not the exit address or location.
The shell commands below keep credentials out of literal command arguments and shell history. Environment variables still belong to the process environment; they are not a substitute for your deployment's secret-management controls.
Proxy configuration
1. Load the endpoint and credentials
- Open a terminal in a new working directory.
- Run the following commands and enter your actual values at the prompts.
- Keep PROXY_URL free of credentials. Enter only the proxy scheme, hostname, and port.
read -r -p 'Proxy URL (scheme://host:port): ' PROXY_URL
read -r -p 'Proxy username: ' PROXY_USER
read -r -s -p 'Proxy password: ' PROXY_PASS
printf '\n'
read -r -p 'Authorized HTTPS target URL: ' TARGET_URL
export PROXY_URL PROXY_USER PROXY_PASS TARGET_URLExpected result: your shell contains four populated variables. Do not print the password to verify it. The Go program checks that the required values exist before making a request.
Keep proxy credentials separate from any target credentials. If the website requires a bearer token, that token belongs to the website request, not the proxy URL. This example intentionally sends no website authentication.
Make configuration validation part of startup. A missing proxy endpoint should stop this client rather than silently route a collection job directly.
Authenticated Go client
2. Create the transport and execute one request
- Save the following program as
main.go. - Leave TLS certificate verification enabled.
- Run
go run main.gofrom the shell containing your exported variables.
package main
import (
"fmt"
"io"
"log"
"net/http"
"net/url"
"os"
"time"
)
func required(name string) string {
value := os.Getenv(name)
if value == "" {
log.Fatalf("missing environment variable: %s", name)
}
return value
}
func main() {
proxyURL, err := url.Parse(required("PROXY_URL"))
if err != nil {
log.Fatal("invalid proxy URL")
}
if proxyURL.Hostname() == "" || proxyURL.User != nil ||
(proxyURL.Scheme != "http" && proxyURL.Scheme != "https") {
log.Fatal("use an http/https proxy URL without credentials")
}
proxyURL.User = url.UserPassword(
required("PROXY_USER"), required("PROXY_PASS"),
)
target, err := url.Parse(required("TARGET_URL"))
if err != nil {
log.Fatal("invalid target URL")
}
if target.Scheme != "https" || target.Hostname() == "" || target.User != nil {
log.Fatal("use an HTTPS target URL without embedded credentials")
}
transport := http.DefaultTransport.(*http.Transport).Clone()
transport.Proxy = http.ProxyURL(proxyURL)
transport.MaxConnsPerHost = 4
transport.MaxIdleConnsPerHost = 4
transport.TLSHandshakeTimeout = 10 * time.Second
transport.ResponseHeaderTimeout = 15 * time.Second
transport.IdleConnTimeout = 90 * time.Second
defer transport.CloseIdleConnections()
client := &http.Client{
Transport: transport,
Timeout: 30 * time.Second,
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
req, err := http.NewRequest(http.MethodGet, target.String(), nil)
if err != nil {
log.Fatal("could not construct request")
}
req.Header.Set("User-Agent", "authorized-collector/2026")
if err := fetch(client, req); err != nil {
log.Print(err)
return
}
}
func fetch(client *http.Client, req *http.Request) error {
resp, err := client.Do(req)
if err != nil {
return fmt.Errorf("request failed: check connection, authentication, and deadlines")
}
defer resp.Body.Close()
const maxBody = 2 << 20
body, err := io.ReadAll(io.LimitReader(resp.Body, maxBody+1))
if err != nil {
return fmt.Errorf("response body read failed")
}
fmt.Fprintf(os.Stderr, "HTTP %d; %d bytes\n", resp.StatusCode, len(body))
if len(body) > maxBody {
return fmt.Errorf("response exceeds 2 MiB limit")
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("unexpected HTTP status: %d", resp.StatusCode)
}
_, err = os.Stdout.Write(body)
return err
}Expected result: the program writes the HTTP status and body length to standard error, then writes an accepted response body to standard output. A successful transport exchange is not enough: inspect the returned content before treating it as collected data.
The example uses a 30-second client timeout, a 15-second response-header timeout, and a 2 MiB body limit. These are example safeguards, not provider specifications or measured performance targets. Adjust them to your authorized workload.
The program deliberately returns generic connection errors so a proxy URL cannot accidentally enter a log through an error string. Production instrumentation should classify underlying errors inside fetch, then emit sanitized error categories. Also return a nonzero process exit status for failed batch jobs so your scheduler can detect failure.
What the transport handles
For an HTTPS destination through an HTTP proxy, Go establishes a CONNECT tunnel. Proxy credentials attached with url.UserPassword let the transport construct proxy authentication. Do not replace that with a manually added Authorization request header.
The transport is safe for concurrent use. Configure it before starting workers, then reuse it. Creating a new transport for every URL discards connection-pooling benefits and makes connection counts harder to control.
Best for: an authenticated Go scraper that needs explicit routing and predictable request boundaries. The limitation is equally clear: net/http fetches HTTP responses; it does not execute browser JavaScript, solve challenges, or validate extracted records for you.
Response verification
3. Verify the exit and the collected content
- Point TARGET_URL at your controlled IP-echo endpoint and run the program.
- Compare the reported exit address with the source address of a direct request from the same machine.
- Change TARGET_URL to the authorized collection URL and run again.
- Check the status, content type, and a page-specific marker before extracting records.
Expected result: the diagnostic response identifies the proxy exit, and the target response contains the content your parser expects. Keep those as separate acceptance checks.
A 200 response can contain a login page, a consent screen, or a challenge instead of the requested record. Status-only monitoring misses that failure. Validate a required field or document marker and record an extraction failure when it is absent.
The example refuses redirects so the first response remains visible. When your target legitimately redirects, inspect the Location value, approve the destination hostname, and implement a bounded redirect policy. Do not automatically follow user-supplied destinations in a collection service.
Record the target hostname, status, elapsed duration, byte count, and validation outcome for every collection job. Exclude credentials, authenticated proxy URLs, cookies, and sensitive response content from routine logs.
Variant: environment-controlled proxy routing
Use environment-controlled routing when deployment configuration should choose the proxy without changing the program. Best for: services whose operators centrally manage outbound proxy settings. Its drawback is that bypass rules can change routing without changing application code.
These are alternative transport configurations, not settings to combine:
| Configuration | Routing behavior | Main limitation |
|---|---|---|
http.ProxyURL(proxyURL) | Returns the configured proxy for requests | Endpoint changes require application configuration changes |
http.ProxyFromEnvironment | Reads proxy settings and bypass rules from the environment | A bypass rule can route a matching target directly |
- Replace
transport.Proxy = http.ProxyURL(proxyURL)with the assignment below. - Remove the explicit proxy parsing and credential requirements from
main. - Supply
HTTPS_PROXYthrough your secret-management system. UseHTTP_PROXYfor HTTP targets if your application permits them. - Review
NO_PROXYbefore testing. Start a new process after changing these settings.
transport.Proxy = http.ProxyFromEnvironmentExpected result: an HTTPS target uses the environment-selected proxy unless a bypass rule applies. Go also treats localhost and loopback destinations specially, so they are not suitable targets for proving environment-based proxy routing.
Use an encoded authenticated proxy URL for the environment value. Do not manually concatenate a raw password containing URL delimiters. The explicit configuration in the main example avoids that manual encoding step and makes the proxy dependency easier to inspect.
Troubleshooting
Proxy authentication fails with 407
Fix the proxy credentials before changing the scraper. Check the username, password, endpoint, and authentication method against the connection details you received. Ensure a secret copied from a file does not include an unintended trailing newline.
For an HTTPS target, a failed CONNECT exchange can appear as a client.Do error rather than a usable response object. Diagnose that path separately from an origin HTTP response.
The target returns 403 or 429
A 403 indicates refusal; it does not identify the reason by itself. Confirm authorization and inspect the response before changing routing. A 429 calls for reduced request pressure and handling of Retry-After when supplied, not immediate repeated retries.
Do not interpret every target error as proxy authentication failure. Repeating valid credentials does not repair a target-side access policy.
Requests time out or connections remain busy
Identify whether the failure occurs during connection setup, TLS negotiation, response headers, or body reading. Increase a deadline only after identifying the stage. A larger timeout does not repair an unreachable endpoint.
Always close the response body. Reading an accepted response through EOF supports connection reuse; oversized responses intentionally stop early in this example, so do not expect their connections to remain reusable.
TLS certificate verification fails
Check the machine clock, trust store, proxy scheme, and target certificate chain. An HTTPS proxy endpoint also requires a valid TLS connection to the proxy itself.
Do not set InsecureSkipVerify to repair the request. That removes certificate verification rather than resolving the trust failure.
The request succeeds but the exit or content is wrong
Retest with the controlled diagnostic endpoint. With environment routing, inspect NO_PROXY and confirm you restarted the process after configuration changes.
If the exit is correct but the page is wrong, inspect redirects and content validation. Proxy routing and usable extraction remain different checks.
Customize your workflow
Add a bounded worker queue only after one request passes every acceptance check. The example's 4-connection limit applies to transport connections, not to a complete job scheduler or a fixed global request rate. Multiplexed protocols and multiple destinations make those different controls.
Give each logical task a cancellation context. Retry only operations that are safe to repeat, use a bounded retry budget, and honor target instructions. Keep the same overall task deadline across attempts so retries cannot extend work indefinitely.
Version the timeout, concurrency, and validation configuration alongside the collector. Keep secrets outside that configuration. When you change routing, rerun both the exit check and the content check before resuming the full queue.
FAQ
How do I configure a golang http proxy with a username and password?
Parse the proxy endpoint with url.Parse, attach credentials with url.UserPassword, and assign http.ProxyURL to the transport's Proxy field. Reuse that transport through an http.Client.
Does an HTTPS target require an HTTPS proxy URL?
No. Go can reach an HTTPS target through an HTTP proxy using CONNECT. Use the scheme specified for the proxy endpoint, not the scheme of the target.
Should I add Proxy-Authorization to every Go request?
Use credentials on the proxy URL and let the HTTP transport handle proxy authentication. Do not put proxy credentials in the target's Authorization header.
Why does my Go proxy request return 407?
A 407 response means proxy authentication is required or was not accepted. Verify the credentials and authentication method; an HTTPS CONNECT failure can surface as a request error.
Does MaxConnsPerHost limit all scraper concurrency?
No. MaxConnsPerHost limits transport connections per host, not the total number of queued tasks or all simultaneous HTTP requests. Use a worker queue or semaphore for application-level concurrency.
Can Go net/http scrape JavaScript-rendered pages?
Go net/http does not execute JavaScript. Use an authorized data endpoint or a browser workflow when the required content depends on script execution.
How can I confirm that Go actually used the proxy?
Request a controlled endpoint that reports the connecting IP address and compare it with a direct request. Repeat this check after changing proxy configuration or environment bypass rules.
Connection reuse is not exit rotation
A reusable Go transport manages connections; the proxy service determines exit assignment. Do not disable keep-alive merely because your collection task needs rotating exits. Verify the service's session behavior with repeated diagnostic requests first.