Proxies for Brand Protection Teams: Evidence Capture

Keep one exit per investigation, save a capture manifest with every page, and know what to do when a proxy session breaks mid-case.

A missing listing can mean two things: the seller removed it, or your collector failed to load it. Keep those outcomes separate. Use a stable exit for a connected investigation, check the market the storefront actually shows, and save enough context for another investigator to reproduce the observation.

This guide is about that last part. It covers which exit to use for which question, how to hold one exit across a seller investigation, a capture script that writes a manifest beside every page, and the decision to make when a session breaks halfway through a case.

Pick the exit for the question

A proxy changes one thing: the network address the target sees. The storefront still reads your cookies, browser language, signed-in account and delivery address, and any of those can decide which market it shows. So the first question is not "which proxy" but "what am I trying to observe, and from where".

Node4's datacenter products (dedicated, shared and the two rotating gateways) run on address blocks Node4 operates in the United States, Italy and Spain. Residential exits come from a partner network covering far more countries, with country, state and city targeting in the username; the residential page lists current coverage. If an investigation needs Japan or Brazil, it needs residential.

Hold one exit for one investigation

A seller investigation is usually a chain: a search result, the seller's profile, three of their listings, maybe their storefront policy page. If the exit changes between those pages, a site that keys anything on the visitor's address may show you a different market halfway through, and the chain stops describing one visitor's view.

Both Node4 gateways accept a session id in the username. Every connection that carries the same id leaves from the same exit until the session ends.

Residential, Germany, one case:
http://USERNAME-country-de-session-case4471:PASSWORD@gw-residential.node4.io:8082

Rotating Datacenter, United States, one case:
http://USERNAME-country-us-session-case4471:PASSWORD@gw-rotating_datacenter.node4.io:8083

Session ids are letters and digits, up to 32 characters, so a case number works well. Two limits shape how you plan a case:

• The pin has a ceiling. On the rotating gateway a session holds its exit for up to 30 minutes, measured from when it is first assigned. Requests during the session do not extend it. Residential sessions are best effort with the same maximum: a hold, not a guarantee.
• A residential exit is someone's connection. If the device behind a pinned residential exit goes offline, requests on that session keep failing until the session window ends. Switch to a new session id, which gets its own assignment from the pool.

Without a session id, each new connection can draw a different exit. That is what you want for independent discovery work, such as checking a list of suspect domains, and it is the wrong setting for a case.

On the rotating gateway you can confirm the pin is working: the proxy's reply to each HTTPS tunnel carries an X-Node4-Sticky header reading opened for a new session, held for a reused one, and none if no session id was parsed. The capture script below works on either gateway because it compares the exit address directly.

A capture script that writes its own manifest

Save the observation before you interpret it. This script fetches one page through a pinned exit, records the exit address before and after the fetch, hashes the body, and writes a manifest with the fields another investigator needs: URL, UTC time, market, account state, exit, content hash and review status.

#!/usr/bin/env bash
# capture.sh URL CASE_ID MARKET
# CASE_ID: letters and digits (it doubles as the proxy session id)
# MARKET:  two-letter country code, e.g. de
# Needs curl, jq and python3 (standard library only, for the IP check).
set -euo pipefail
url="$1"; case_id="$2"; market="$3"
: "${N4_USER:?}"; : "${N4_PASS:?}"
command -v python3 >/dev/null || { echo "capture.sh: python3 is required" >&2; exit 1; }

proxy="http://${N4_USER}-country-${market}-session-${case_id}:${N4_PASS}@gw-residential.node4.io:8082"
captured=$(date -u +%Y-%m-%dT%H:%M:%SZ)
dir="captures/${case_id}/${captured//:/-}"
mkdir -p "$dir"

# Prints the exit address, or "unknown" if the probe failed or returned anything else.
exit_ip() {
  local ip
  ip=$(curl -fs --connect-timeout 10 --max-time 20 --proxy "$proxy" https://api.ipify.org) || ip=""
  python3 -c '
import ipaddress, sys
try:
    print(ipaddress.ip_address(sys.argv[1].strip()))
except ValueError:
    print("unknown")' "$ip"
}

exit_before=$(exit_ip)
codes=$(curl -s --connect-timeout 10 --max-time 60 --proxy "$proxy" \
  -D "$dir/headers.txt" -o "$dir/body.html" \
  -w '%{http_code} %{http_connect}' "$url" || true)
read -r status connect_status <<< "${codes:-000 000}"
exit_after=$(exit_ip)

body_sha=$(sha256sum "$dir/body.html" 2>/dev/null | cut -d' ' -f1 || true)

jq -n \
  --arg case "$case_id" --arg url "$url" --arg market "$market" \
  --arg captured "$captured" --arg state "logged-out, clean profile" \
  --arg before "$exit_before" --arg after "$exit_after" \
  --arg status "${status:-000}" --arg connect "${connect_status:-000}" \
  --arg sha "${body_sha:-}" \
  '{case: $case, url: $url, market: $market, captured_utc: $captured,
    account_state: $state, exit_before: $before, exit_after: $after,
    exit_held: (if $before == "unknown" or $after == "unknown" then null
                else $before == $after end),
    proxy_connect_status: ($connect | tonumber),
    http_status: ($status | tonumber), body_sha256: $sha,
    outcome: "unreviewed", reviewer: null}' > "$dir/manifest.json"

cat "$dir/manifest.json"

A few choices in it are deliberate:

• The exit is checked on both sides of the fetch. With a session id, all three connections should leave from one address. Matching addresses before and after support the case that the page came through the same exit; they do not prove the route of the fetch in between. If exit_held is false, the page may have been served to a different exit than the one you recorded.
• A failed probe is unknown, not a match. exit_ip uses curl -f, so an HTTP error from the echo service fails instead of being saved as an address. The body is then parsed with Python's ipaddress module, so only a valid IP address is recorded, written in its standard form so both sides compare cleanly; an empty reply, an HTML page or something like :::: becomes unknown. If either side is unknown, exit_held is null: the hold was not checked.
• A failed fetch still produces a manifest. http_status of 0 means the request never completed, which is a collection failure, not evidence that anything was removed. proxy_connect_status is the proxy's answer to the tunnel request, kept apart from the target's: a 407 there is a credential problem and a 502 a refused route, while a 403 in http_status is the target refusing you.
• outcome starts as unreviewed. A person sets it later. The script never decides that a listing infringes anything.
• The hash covers the body as saved. It shows the file has not changed since capture. It says nothing about who published the page.

curl saves the raw response. If the listing is rendered by JavaScript, the body may be an empty shell, so pair this with a browser capture (screenshot plus rendered HTML) through the same proxy string. Response headers can contain cookies, so store the capture folder with the same access controls as the case file.

When the session breaks mid-case

Sooner or later a manifest comes back with exit_held: false, or a residential session starts failing because its device went offline. The case is not lost, but the captures need sorting before anyone relies on them.

1. Mark the break. Note which captures were taken before the exit changed and which after. Do not merge them silently into one sequence.
2. Check whether it matters for this target. Compare the market signals in the pages from both sides of the break: currency, language, delivery country, the seller's displayed region. If they match, record that comparison and continue.
3. If they differ, restart the chain. Open a new session id (for example case4471b), go back to the first page of the sequence, and capture it again. Keep the interrupted captures in the file, labeled as interrupted.
4. Plan long cases in chunks. A case that will run past the session ceiling should be split at a natural point, such as after the seller profile, with a fresh session for each chunk and the split noted in the case log.

Reading an empty result

The most expensive mistake in brand monitoring is reporting a takedown that did not happen. Give every observation one of these outcomes before it reaches a report:

Only the last row supports "removed". Everything above it means you did not see the listing, which is a different statement.

FAQ

Do brand protection teams need residential proxies?

For marketplace and social listings in a specific country, usually yes, because many of those sites treat hosting networks differently from home connections. For checking domains, redirects and reseller sites in the US, Italy or Spain, datacenter exits are often enough. Run your approved reference pages through both and compare what each returns.

Can a proxy show me exactly what a local shopper sees?

It gives you a local network address, which is one input. What a shopper sees also depends on their account, cookies, language and delivery address. Record those settings in the manifest so the observation can be repeated.

Does a proxy give us permission to collect from a site?

No. Collection scope, site terms and access to anything behind a login are decisions for your legal and enforcement team. The proxy only changes the route.

Related guides

• Sticky sessions explained
• How accurate proxy location really is
• Proxies for ad verification
• Residential country and city targeting
• Proxies for web scraping agencies

See pricing for current residential and datacenter plans.