Proxies for Real Estate Data: One Row per Property
Keep one row per property across portals: listing IDs, duplicate detection, update times, and telling a removed listing from a refused request.
If you track listings across property portals, the hard part is keeping one row per property while listings get reposted, re-priced and withdrawn. Proxies decide where your requests come from and let a crawl run from the right country. They don't tell you that the apartment on two portals is the same unit, or that a listing disappeared because it sold rather than because your request was refused. Both of those are decisions your pipeline makes, and the proxy setup should support them rather than blur them.
Where a portal or MLS offers a licensed feed, use it. Crawl only sources you are permitted to collect, within their terms and rate limits.
One property, four captures
For example, suppose a crawler collects one Madrid apartment from two portals over a week. The captures might look like this (the listing IDs, prices and times are illustrative):
| Capture | Source | Listing ID | Normalized address | Beds / baths / m² | Price | Source "updated" | Result |
|---|---|---|---|---|---|---|---|
| 1 | Portal A | 7731902 | CALLE MAYOR 41, 3B, 28013 MADRID | 2 / 1 / 74 | 425,000 EUR | Oct 1, 10:12 | Active |
| 2 | Portal B | B-55120 | CALLE MAYOR 41, 3B, 28013 MADRID | 2 / 1 / 74 | 429,000 EUR | Sep 29, 18:40 | Active |
| 3 | Portal A | 7731902 | same | same | 415,000 EUR | Oct 4, 08:05 | Active, price change |
| 4 | Portal A | 7731902 | n/a | n/a | n/a | n/a | HTTP 200, "listing no longer available" |
What the pipeline should conclude:
- Captures 1 and 2 are one property with two listings. The source listing IDs differ, so the property key cannot be a listing ID. Use the normalized address plus unit, confirmed by matching beds, baths and floor area. Never merge on price; two units in one building can share it.
- Capture 3 is a price event, because the source's own update time is newer than the stored one. If a capture arrives with an older "updated" time than you already hold (a cached page, a mirror, a slow replica), ignore its price instead of reverting your row.
- Capture 4 is a removal, but only after confirmation. The portal returned a normal page saying the listing is gone. That differs from a 403, a challenge page or a timeout, which tell you nothing about the property. Record those as "unknown, keep last known state", and re-check from a different exit before setting the status to withdrawn.
The difference between "removed" and "blocked" matters most in days-on-market and absorption figures. A crawler that marks every refused request as a withdrawal reports a market that is selling faster than it is.
Exits for local portals
Most property portals are national, and some show different results, or refuse entirely, depending on where the request comes from.
Node4's datacenter products run on Node4-owned IP blocks in the United States, Italy and Spain, so Italian and Spanish portals as well as US sites can be reached from in-country datacenter addresses. A fixed dedicated datacenter address suits a nightly crawl of one portal where you want each run to look like the last. For other countries, residential proxies use a partner network in 160+ countries with country and city targeting in the username. Some portals treat datacenter and household addresses differently, so test a sample search from each before committing.
Keeping a search session together
Portal search results are usually paginated, and some bind the result cursor to session state. Request page seven from a new exit and you may get page one again, or the same listings in a different order, which shows up later as phantom duplicates and missing properties.
Rotating gateways assign a new exit to each HTTPS tunnel (and to each plain-HTTP request); requests through the same open tunnel keep it. To hold an exit across a whole search, put a session ID in the username:
curl -x "http://USERNAME-country-it-session-re3301:PASSWORD@gw-residential.node4.io:8082" https://api.ipify.orgOn Rotating Datacenter and Rotating Unmetered plans, that ID keeps one exit for up to 30 minutes from when it is first assigned, however busy the session is. Residential sessions are best effort, with a 30-minute ceiling, and can end sooner if the device behind the exit goes offline. Split a large search into smaller queries (by district or price band) so each one finishes inside its session, and if the exit changes partway, rerun that query from page one. The sticky versus rotating sessions guide covers how to detect the change.
Detail pages for listings you already know are independent fetches. They don't need a shared session and can spread across exits.
Re-crawl cadence and cost
Not every listing needs checking every day. A practical schedule:
- Active listings with a recent price change: check daily.
- Stable active listings: every two or three days.
- Listings in "unknown" state after a refused request: re-check soon, from a different exit.
- Confirmed removals: one follow-up a week later to catch relistings, then stop.
That schedule, not the size of the market, sets your request volume. Rotating Datacenter is billed per GB (rotating prices), as is residential (residential prices). For portals with image-heavy pages and a hard-to-forecast transfer volume, Rotating Unmetered is billed by concurrent connections with no per-GB charge (connection-based pricing).
Where to start
Pick one portal and one district. Collect it by hand and through the proxy, build the property key from normalized address and unit, and check that every listing lands in exactly one row. Then add the second portal and confirm the overlaps merge. The Puppeteer and Playwright proxy guide helps if a portal renders its listings with JavaScript, and the recruitment guide applies the same one-row-per-entity deduplication to job postings.