Proxies for Market Research: Defensible Regional Prices

Collect regional prices and availability you can defend: an exit per market, one session per browsing task, and rejecting records from the wrong market.

If you're comparing prices or product availability across countries, each request needs the right location and consistent account settings. A proxy can change the IP address a site sees. It won't set your delivery address, clear your cookies, or prove that the page belongs to the market you're studying. This guide covers the three decisions that make the difference: which exit serves each market, how long one browsing task keeps its exit, and which records you accept.

A record that looks fine and is wrong

Start with the failure, because it is the one a dashboard of HTTP status codes will not show you. For example, suppose a collector requests a British grocery product page through a British exit and stores this record:

{
  "task_id": "gb-grocery-sku-88317",
  "requested_market": "GB",
  "exit_country": "GB",
  "http_status": 200,
  "price": "4.99",
  "currency_displayed": "USD",
  "delivery_region_selected": "US",
  "page_language": "en-US",
  "validation_result": "rejected: wrong market (expected GBP and a GB delivery region)"
}

Everything at the network layer worked: the exit is in the United Kingdom, the page loaded and the parser found a price. The record is still useless. A typical cause is a worker that reused the cookie jar from a previous US run, so the retailer honored the stored delivery address over the IP address and showed a US price on a British request.

The fix is in the collector, not the proxy:

  • Give each market its own cookie jar, and never share one across markets.
  • Set the delivery region or store explicitly at the start of each session, then read it back from the page.
  • Accept a record only when the displayed currency, the delivery region and the product identifier all match the study specification.

Keep rejected records with their reason instead of discarding them. "Wrong market", "not in this catalog", "challenge page" and "parser failure" mean different things. A product missing from the Spanish catalog is a finding; a challenge page is not evidence of anything, and counting it as "unavailable" quietly corrupts the availability figures you report.

Matching exits to the markets in your sample

Write the market list before you compare providers, then check each market against what the product actually covers.

Node4's datacenter products (shared, dedicated and Rotating Datacenter) run on Node4-owned IP blocks in the United States, Italy and Spain. If your study covers those three markets, a datacenter exit is a straightforward fit. For anything else, such as the United Kingdom, Germany or Japan, the option is residential proxies: exits from a partner network in 160+ countries, chosen with a country segment in the username. The country and city targeting guide has the full syntax, including how a location with no available exits is refused rather than silently swapped for another country.

Some retailers treat datacenter and household addresses differently, showing a reduced catalog or a challenge to one and not the other. Run your manual sample through both before committing a market to datacenter exits.

Study shapeProductWhy it fits
The same products in US, Italian or Spanish stores, checked on a scheduleDedicated datacenter (prices)One fixed address per market keeps every run comparable
Many independent product pages in those three countriesRotating Datacenter (prices)A fresh exit per task, billed per GB
Markets outside the US, Italy and SpainResidential (prices)Country targeting across the partner network
Large recurring crawls where data volume is hard to predictRotating Unmetered (prices)Billed by concurrent connections, with no per-GB charge

Keeping one browsing task on one exit

A category page followed by its product pages is one observation. If the exit changes halfway through, you can end up with a listing from one regional context and details from another.

The gateways pick an exit for each HTTPS tunnel or SOCKS5 connection, and for each plain-HTTP request. Requests sent through one open HTTPS tunnel keep the same exit, and a session ID in the username holds one exit across connections. A British residential session for one task looks like this:

curl -x "http://USERNAME-country-gb-session-mr0412:PASSWORD@gw-residential.node4.io:8082" https://api.ipify.org

How long that pin lasts depends on the product:

  • On Rotating Datacenter and Rotating Unmetered plans, a session ID holds one exit for up to 30 minutes, measured from first assignment. A busy session does not extend itself.
  • On residential plans, sessions are best effort, with a 30-minute ceiling. A pin can fail sooner if the device behind the exit goes offline; after repeated failures, switch to a new session ID.

So size each task to finish well inside its session. One market's category walk, bound to one session ID and one cookie jar, is a sensible unit. If the exit changes mid-task, restart the walk with a new session ID instead of stitching two halves together. The sticky versus rotating sessions guide covers detection and restart in more detail.

Logging so a disputed number can be explained

A client will eventually ask why the Italian price for one product looks wrong. You need to answer from the log, not from memory. Record, for every observation:

  • task_id, requested_market and captured_at
  • the exit address and the country the exit reported
  • the session ID and the cookie jar used
  • currency_displayed, delivery_region_selected and the product identifier the page showed
  • the extraction version and the validation result with its reason

Store proxy credentials elsewhere. A log line should let someone rerun the observation, not reuse your account.

Keep a small manual control sample for each market: a researcher opens the same ten products in a normal browser with the market's settings and records what they see. When the automated and manual results disagree, check the session's region settings and the parser before you change anything about the proxies. In the example record above, the likely cause is a stored cookie, not the exit.

Where to start

Pick one market and ten products. Collect them by hand, then through the proxy with a dedicated cookie jar and an explicit delivery region, and compare the two sets using the acceptance rule above. Once that market matches, add the next one. For the recurring version of this work, price monitoring with proxies covers scheduling and change detection, and the rotating proxies for web scraping guide covers session design for larger crawls. If your study is fares or ad placements rather than retail prices, the guides for travel fare aggregators and ad tech teams cover what those records need. A reseller pricing a product from the same kind of listing data, rather than studying a market, has a different acceptance rule and account-separation requirement: see proxies for e-commerce resellers.