Price Monitoring with Proxies: Getting Prices You Can Trust

Personalised, geo-varied and cached prices make single-IP scrapes wrong. How to collect competitor prices that reflect what buyers actually see.

Price monitoring exists to answer one question: what does a buyer see when they look at this product right now? Repricing engines, buy-box strategies and margin reports all sit downstream of that answer, which means a wrong answer is worse than no answer at all. A repricer fed a stale or personalised competitor price will cheerfully undercut a number nobody is actually charging, and it will do so at scale, automatically, until someone notices the margin report.

This guide is about the ways price data goes wrong before it ever reaches your database, and how vantage point, product matching and observation design fix them. The proxy is the smallest part of the system. It is also the part that determines whether everything downstream is measuring reality.

The price on the page is an answer to a question you did not ask

The naive model of an e-commerce page is that it has a price and your scraper fetches it. Modern retail does not work that way. The number rendered into the page is computed per request, and several inputs to that computation come from the request itself:

• Geography. The same retailer shows different figures to visitors from different countries, and sometimes different regions of one country: different list prices, different promotions, different delivery surcharges folded into or out of the displayed number.
• Session state. Logged-in shoppers with loyalty status, items in a basket, or a history of abandoning carts can be quoted differently from anonymous first-time visitors. Some retailers move a price after you have viewed it twice.
• Device and channel. App-only pricing, mobile-web promotions and marketplace-specific offers mean the "same" product carries several simultaneous prices depending on where you look.
• Experiments. Retailers A/B test price presentation constantly. A fraction of visitors will see a test cell (a different bundle, a struck-through anchor figure, a coupon banner), and if your scraper lands in that cell, you record the experiment, not the market.

The consequence is that a single scraping server in one datacenter does not observe "the price". It observes the price served to one unusual visitor: an anonymous, cookieless client on a hosting-range IP, which is close to the least representative shopper a retailer ever sees. Some sites respond to exactly that profile by serving a deliberately degraded page, a default national price, or a block page dressed up as an out-of-stock notice.

Fixing the vantage point is what proxies are for here. Observing from residential exits in the buyer's own market, with the country pinned per request, produces the page a real shopper in that market receives. Observing the same product from two or three distinct exits in the same market tells you whether the price is stable or you have landed in an experiment. For catalog-scale coverage across many retailers, rotating access spreads the requests so no single address builds up a scraping profile.

Product matching: the identity problem is harder than the scraping

Before a price comparison means anything, both sides must refer to the same thing, and product identity across retailers is genuinely messy:

• SKUs are private. A retailer's SKU is its own inventory key, not a shared identifier. Two retailers' SKUs for the identical item share nothing.
• GTINs help until they don't. Barcode-level identifiers (EAN, UPC, ISBN) are the closest thing to a universal key, but marketplaces are full of listings with missing, wrong or deliberately reused GTINs, and manufacturer variants sometimes share one.
• Variants multiply. Size, color, storage capacity, regional plug type: each is a distinct purchasable item with a distinct price. Matching "the product" without matching the variant produces comparisons between a 128 GB and a 256 GB model that look like a competitor undercutting you by a third.
• Bundles and pack sizes lie by omission. A three-pack priced 40% above your single unit is more expensive per unit and will appear cheaper in a naive comparison. Normalize to unit price where the category allows it, and store the pack quantity alongside every observation so the normalization can be audited later.
• Marketplace sellers are not the retailer. On marketplace platforms, the number you scrape is whichever offer currently holds the featured position. Track the offer's seller identity, not just the figure. A price that "dropped" because a different seller with a worse delivery promise won the slot is a different fact than the incumbent cutting price.

Get the matching layer wrong and the proxy layer is irrelevant: you will be comparing perfectly-collected prices of different things.

Currency, tax and the display trap

Cross-market comparison adds a second identity problem: the identity of the number. A figure scraped in Germany typically includes VAT because consumer-price display rules there require it; the equivalent US figure excludes sales tax because tax is added at checkout by delivery address. Comparing the two raw is a built-in error the size of the tax rate.

Decide, per market, what the collected number represents (tax-inclusive shelf price, tax-exclusive base, delivered price with shipping), and record that decision with the data. Store the currency code with every observation and convert at analysis time with a dated exchange rate, never at collection time, so a currency movement is not silently baked into a "price change". And beware sites that geo-detect and convert prices for foreign visitors: a converted price is the site's estimate at its own chosen rate, not the market price locals pay. This is another argument for in-market exits: a shopper in Madrid sees euros priced for Spain, so your Spanish observations should come from Spanish exits, not from a US address with a currency banner. Before committing a market to a schedule, it is worth checking that the coverage supports the granularity you want: the pages for Spain, Germany and the United States each list the cities currently targetable there, and the US in particular is really several markets, since tax, shipping cutoffs and stock differ by state.

The silent failure: recording a cached or personalised page as the market

The failure mode that deserves the most paranoia is not being blocked. Blocks are loud. The dangerous outcome is a page that renders perfectly, parses cleanly, and carries a number that is not the current market price, because it came from a cache, or because it was personalised to your scraping profile.

Caching happens at several layers you do not control: the retailer's CDN serving a stale edge copy, an intermediate cache keyed without the pricing cookie, or the retailer's own page generator running behind its pricing engine's update cycle. Personalisation is the mirror image: the page is fresh, but fresh for the visitor you appeared to be.

Neither failure announces itself, so the collection system has to be designed to catch it:

• Scrape prices from the page's structured data and its visible DOM, and compare. Product schema markup is often generated on a different cadence from the rendered price block. When the two disagree, the page is telling you at least one of them is stale; flag the observation instead of picking one.
• Run canary products. Keep a handful of items whose price you control or verify manually, collect them through the same pipeline, and alert when the pipeline reports a number you know is wrong. This is the single highest-value integrity check available.
• Double-observe before believing a change. When a price moves, re-fetch through a different exit before recording the change as fact. A real repricing shows from both vantage points; a cache artefact or experiment cell usually does not.
• Watch for impossible stability. A product whose price has not moved in weeks while its rivals reprice daily is at least as suspicious as one that swings. Perfect flatness is what a cached page looks like.

Stale versus changed: the data model does the work

The design decision that separates robust price intelligence from a brittle scraper is refusing to store "the current price" as a mutable field. Store observations: product identity, variant, seller, raw price string as rendered, parsed value, currency, tax basis, exit country, timestamp, and a fetch outcome. Derive "current price" and "price changed" as queries over observations.

This buys you three things. First, failed or suspect fetches stop masquerading as data: an observation with a block outcome is recorded and excluded, instead of the previous price silently persisting as if reconfirmed. Distinguishing "we saw the same price again" from "we have not successfully looked recently" is the whole difference between stale and unchanged, and only an observation log can make it. Second, you can re-parse history when a retailer changes its page layout, because the raw evidence is retained. Third, change detection becomes debounceable: require two consecutive confirming observations before emitting a change event, and flapping caused by experiments or offer rotation stops paging anyone.

Scheduling: frequency should follow volatility

Fetching every product at the same fixed interval wastes most of your bandwidth on items that never move and under-samples the ones that move hourly. Let the observation history drive the schedule: products with recent changes, promotion flags or marketplace offer churn get sampled more often; the long tail that repriced twice last quarter gets sampled daily or weekly. Around known volatility events (sale seasons, a competitor's announced promotion), temporarily promote whole categories.

This is also where request-rate discipline lives. A monitoring workload runs continuously against the same set of retailers, which makes it more detectable over time than a one-off crawl. Spread requests across the day rather than firing the whole catalog at the top of the hour, randomize ordering, and keep per-retailer concurrency low. If a retailer starts serving blocks, backing off is not politeness; it is protecting the vantage point your data quality depends on. The mechanics of spreading load across exits are covered in the IP rotation guide, and how many proxies you need works through sizing the pool against catalog size and fetch frequency.

Choosing the access layer

Not every retailer requires the same machinery, and paying for residential bandwidth to scrape a site that happily serves datacenter ranges is money spent on nothing. A workable ladder:

• Datacenter proxies for retailers that do not classify hosting IPs, often smaller shops and price-comparison-friendly sites. Cheap per request, fast, and adequate for a surprising share of a typical catalog. See the datacenter option for when this tier fits.
• Rotating residential exits for the protected majors, where a consumer-looking address in the right country is a precondition for seeing consumer pricing at all.
• Sticky sessions for the subset of flows where a price only appears after navigation (adding to basket to reveal a checkout price, for instance), where the session must hold one exit across several requests. The trade-offs are covered in sticky versus rotating sessions.

Per-product-per-market cost is the metric to watch: bandwidth-billed residential collection rewards lean fetches, so strip images and third-party scripts from your requests where the price survives without them.

Price intelligence is one of a family of in-market observation workloads that share this access layer while differing in what corrupts them. Ad verification faces a counterparty with an incentive to show the observer something flattering; rank tracking fights contamination from the measurer's own search footprint. Price monitoring's enemy is subtler than either: nobody is deceiving you on purpose; the infrastructure is just answering a differently-shaped question than the one you meant to ask.

The part that is your call

A note on boundaries. Collecting publicly displayed prices is a long-established commercial practice (retailers do it to each other constantly), but the terms of service of individual sites, and the law of the places you and they operate, are not uniform and not this article's to interpret. Read the terms of the sites you monitor, honor explicit technical signals where you decide they apply, and take advice where the stakes justify it. What this guide describes is technical practice; whether and where to apply it is a judgment that stays with you.

Start small: one market, one retailer set, canary products from day one, and an observation log rather than a price field. The free tier is enough to prove the pipeline end to end (matching, parsing, double-observation) before scaling a catalog onto it.