Proxies for Alt Data Teams: Protect the Time Series

Keep collection failures out of an alternative-data series. Log route and parser version per observation, and tell missing apart from absent and stale.

A collection failure should not become a market signal. Record the requested market, exit, parser version and response status with each observation, and keep failed access separate from a confirmed missing listing. Choose proxies by the sources and session behavior your dataset needs.

For a hedge fund or an alternative-data vendor, the proxy is a small part of the stack, but it sits exactly where a silent error turns into a false signal. A run that could not reach a retailer looks, in a naive pipeline, like a retailer that stopped listing products. This guide is about preventing that.

Missing, absent and stale are different values

Every observation in a web-collected series should end up in one of four states, and the series should treat each one differently.

State	What it means	Evidence required	What the series does
Present	The item was seen in the intended market	Valid page, content checks passed, item found	Record the value
Absent	The source itself says the item is gone	Valid page that explicitly shows removal, or a not-found page this source is known to use for removed items	Record the absence
Not observed	You did not get a usable view of the source	Transport error, proxy refusal, challenge page, wrong market	Leave a gap; never record absence
Stale	The last good value is being carried forward	A prior Present observation and its timestamp	Carry the value with its age, and stop after a set limit

The dangerous transition is Not observed being written as Absent. A job-listings series that loses access to one employer for a week shows a hiring freeze. A price series that falls back to the wrong country shows a price change. Both look like data.

The observation record

Store one row per observation, with enough context that an analyst can reclassify it later without recollecting:

source_id            retailer-de-007
entity_id            sku 4411-208
requested_market     DE
route                residential, country de, no session
exit_ip              recorded at fetch time
collected_at_utc     2026-10-06T04:12:09Z
proxy_connect_code   200
http_status          200
content_checks       market=DE currency=EUR page_type=product challenge=no
content_valid        yes
parser_version       retailer-de@3.4.1
state                present
value                129.90 EUR

route and parser_version are the fields teams most often leave out, and they are the two that explain most breaks in a series. If either changes, the series has a new collection regime from that timestamp, and the change belongs in the dataset's release notes beside any source change.

Keep the raw response for at least as long as you might need to reclassify, within your retention policy. A classifier fix applied to stored responses repairs history; a classifier fix with nothing stored only repairs the future.

Choose routes that fail loudly

For a time series, a route that refuses is better than a route that substitutes. Two Node4 behaviors matter here:

A country you ask for, or an error. If you request a country the rotating gateway has no exits in, the request is refused with a 502 rather than sent out through another country. The residential gateway also refuses a location it cannot serve instead of substituting one. A refused request is a clear Not observed; a silently substituted country would have been a wrong-market Present.
Sessions only where the source needs them. Without a session id each new connection can draw a different exit, which suits independent product-page fetches. Paginated listings and multi-step searches need a session id so every page comes from one exit. On the rotating gateway a session holds its exit for up to 30 minutes from first assignment; residential sessions are best effort with the same maximum. Longer runs need planned breaks.

On coverage: Node4's datacenter exits (dedicated, shared, Rotating Datacenter and Rotating Unmetered) are in the United States, Italy and Spain, on address blocks Node4 operates. Residential covers far more countries through a partner network, with country, state and city targeting. A source universe that spans markets outside those three countries needs residential for those markets, and the route field records which is which.

For heavy, steady collection on datacenter exits, Rotating Unmetered is priced by concurrent connections rather than by gigabyte, which makes the cost of a daily run predictable. Rotating Datacenter draws the same exits and bills per gigabyte, which suits smaller or irregular jobs. Current rates are on the pricing page.

When you test a route, capture the proxy's answer separately from the target's. curl reports both:

curl -s --connect-timeout 10 --max-time 60 -o page.html \
  --proxy "http://${N4_USER}-country-de:${N4_PASS}@gw-residential.node4.io:8082" \
  -w 'proxy_connect=%{http_connect} http_status=%{http_code}\n' \
  "https://retailer.example/product/4411-208"

For an HTTPS target, proxy_connect is the proxy's response to the tunnel request and http_status is the target's. A 502 in the first is a refused route; a 403 in the second is the target refusing you. Both are Not observed, but they have different fixes, so store them separately.

A reconciliation example

A retailer series shows SKU 4411-208 in Germany on Monday and Tuesday, then nothing on Wednesday and Thursday, then present again on Friday. Before anyone reads that as a two-day stock-out, check the observation rows:

Wednesday: proxy_connect_code 200, http_status 200, but content_checks shows challenge=yes. The page was a bot check. Not observed.
Thursday: http_status 200, page valid, market DE, and the product page shows the retailer's own "no longer available" notice. Absent.

So the series has one real absence (Thursday), not two, and Wednesday is a gap. If Wednesday's value is carried forward as Stale, it carries Tuesday's timestamp so a model can discount it.

The same logic as code, run over stored observations:

def classify(obs: dict) -> str:
    if obs.get("transport_error") or obs.get("proxy_connect_code") not in (None, 0, 200):
        return "not_observed"          # timeout, proxy refusal (e.g. 502), auth failure
    status = obs.get("http_status")
    if status not in (200, 404, 410) or obs.get("challenge"):
        return "not_observed"          # 403, 429, 5xx, or a bot check served as 200
    requested, seen = obs.get("requested_market"), obs.get("market")
    if not requested or not seen or seen != requested:
        return "not_observed"          # market unknown, or loaded in the wrong market
    if obs.get("content_valid") is not True:
        return "not_observed"          # page not validated against this source's templates
    found, gone = obs.get("item_found") is True, obs.get("explicit_absence") is True
    if found and gone:
        return "not_observed"          # contradictory signals: check the parser
    if found and status == 200:
        return "present"
    if gone:
        return "absent"                # this source's own removal notice or not-found page
    return "not_observed"              # generic 404, or neither found nor removed

explicit_absence is set by the source's parser only when the page matches that source's known removal notice or not-found template. A bare 404 status is not enough: sites return 404 for moved URLs, expired search pages and blocked requests too. content_valid is that parser's verdict that the page is one of the source's known templates, and market is the market the page itself shows (domain, currency, locale). A missing market on either side, or a page the parser did not validate, stays a gap: two empty market fields are not a match.

The last line matters most: when the evidence is ambiguous, the default is a gap, not an absence.

What procurement and compliance will ask

A fund's data review usually wants to know where a dataset came from and whether it could have been distorted by how it was collected. With the observation record above you can answer the second part directly: every value carries its route, market check, parser version and state, and every regime change is dated.

The first part is outside the proxy layer. Source terms, licensing, personal data and the intended research use are decisions for your compliance team. A proxy changes the network route of a request. It does not grant permission to collect anything.

FAQ

Should alternative-data collection use residential or datacenter proxies?

It depends on the source and the market. Sources in the US, Italy or Spain that accept datacenter traffic can use datacenter exits; other markets, and sources that treat hosting networks differently, need residential. Run both against the same sources and compare valid observations, not request counts.

How do I stop a proxy outage from looking like a market event?

Record the proxy's response and the target's response separately, classify every observation as present, absent, not observed or stale, and never write a failed or ambiguous fetch as absent.

Does changing proxy provider break a time series?

It can. Treat a route change like a parser change: date it, note it in the release log, and compare overlapping runs on the old and new routes before splicing the series.

Related guides

See Rotating Datacenter and Rotating Unmetered for the gateway products.