Proxies for Travel Fare Aggregators: Comparable Fares
Two fares are comparable only when route, dates, passengers, baggage, currency and point of sale match. How to collect them that way through proxies.
If you collect fares to show travelers the cheapest option, two prices are only comparable when they describe the same trip: same route and dates, same passenger mix, same cabin and fare brand, same baggage, in a known currency from a known point of sale. A proxy controls one of those, the point of sale a booking site infers from your IP address. Everything else is your collector's job, and it is where most bad comparisons come from.
Where an airline or online travel agency offers an affiliate API, a GDS connection or an NDC feed, use it. Web collection fits the sources you are permitted to read that have no feed, and it should follow each site's terms and rate limits.
When two fares are actually comparable
For example, suppose a collector captures two fares for the same search, London Heathrow to Madrid, out 14 November and back 18 November, one adult. The figures below are illustrative:
| Field | Fare A | Fare B |
|---|---|---|
| Exit and point of sale | GB exit, UK site edition | ES exit, Spanish site edition |
| Fare brand | Economy Basic | Economy Classic |
| Base fare | 62.00 | 58.00 |
| Taxes and carrier charges | 41.30 | 39.10 |
| Total shown | 103.30 GBP | 97.10 EUR |
| Checked bag | Not included, 35.00 GBP to add | One 23 kg bag included |
| Seat selection | Paid | Paid |
| Captured | 09:02 UTC | 09:03 UTC |
A naive comparison says Fare B is cheaper because 97.10 is less than 103.30. That comparison mixes currencies, fare brands and baggage. To compare them you need three steps:
- Convert with a recorded rate. Store the exchange rate and its source at capture time, not at report time.
- Normalize to the traveler's need. For someone checking a bag, Fare A costs 138.30 GBP. For someone traveling with only a cabin bag, the 103.30 figure stands.
- Label the fare brand. Basic and Classic carry different change and refund rules. Show them as different products, not as one price that moved.
A record without taxes, baggage status or fare brand cannot be compared, so mark it "incomplete" instead of putting it in the ranking. The capture times matter too: fares move quickly, so only compare captures taken within a short window of each other, and store the window you used.
The fields worth storing for every fare are the search parameters (origin, destination, dates, passengers, cabin), the fare brand, base, taxes, total, currency, baggage and seat inclusions, the site edition, the exit country, and captured_at.
Point of sale, site editions and exits
Booking sites decide what to show from several signals at once: the IP address, the site edition in the URL or domain, a currency or country selector, and cookies from earlier visits. Change the exit and leave a cookie from the Spanish edition in place, and you may get a British IP address looking at euro prices. Set the edition and currency explicitly at the start of every search, use a fresh cookie jar per point of sale, and read the currency back from the results page before you accept anything.
Node4's datacenter exits are on Node4-owned IP blocks in the United States, Italy and Spain, which covers those three points of sale. For other markets, residential proxies use a partner network in 160+ countries, selected with a country segment in the username. Booking sites often treat datacenter traffic differently from household traffic, so test both on a sample of your real searches before choosing one per source.
Holding a search on one exit
A fare search is rarely one request. Search results, the selected flight, the fare brand options and the baggage step usually form one flow, and many sites tie the result set to a session token. If the exit changes partway through, the site may restart the search or return a different price for the "same" flight.
Rotating gateways choose a new exit for each HTTPS tunnel (and each plain-HTTP request), and keep it for every request through that tunnel. To keep a whole search on one exit, add a session ID to the username:
curl -x "http://USERNAME-country-es-session-fare7c21:PASSWORD@gw-residential.node4.io:8082" https://api.ipify.orgThe pin length is product-specific. On Rotating Datacenter and Rotating Unmetered plans a session ID keeps one exit for up to 30 minutes from first assignment, and continued activity does not extend it. Residential sessions are best effort, with a 30-minute ceiling, and can end sooner if the device behind the exit goes offline. A fare search takes seconds to a few minutes, so one session ID per search is the right unit; when a session ends mid-search, run the search again with a new ID rather than combining results from two exits. Sticky versus rotating sessions shows how to detect that change.
Independent searches, such as the same route on different dates, can each use their own session or none at all.
Load and cost
Fare pages are heavy, and repeating identical searches adds load to the source without adding information. Cache results for the window you consider fresh, deduplicate identical queries across users, and cap concurrent searches per source.
For billing, the question is how predictable your data volume is. Rotating Datacenter is billed per GB (rotating prices). Residential is also billed per GB (residential prices). If you run a large search schedule whose transfer is hard to forecast, Rotating Unmetered is billed by concurrent connections with no per-GB charge (connection-based pricing). The proxy calculator estimates the bandwidth and concurrency a search run needs from its request count and average page size.
Where to start
Take five routes your users search most. Collect each from two points of sale by hand, fill in the comparison table above, then repeat through the proxy with one session per search and confirm the fields match. The price monitoring guide covers scheduling and change detection once single searches are reliable, and the market research guide shows the same accept-or-reject approach applied to retail prices.