Proxies for Recruitment Agencies: Deduplicate Job Posts

Collect job postings from the right country and turn reposts into one opening, deduplicated by company, location and a reliable posted date.

If you aggregate job postings for sourcing, market mapping or client reports, the same opening shows up on the employer's careers page, two job boards and an agency repost, each with a slightly different title, location and date. Proxies let you query job boards from the country whose listings you need. The useful output, though, is one row per real opening with a posted date you can defend, and that comes from deduplication, not from the network.

Keep the collection to job postings and company pages you are permitted to read. Candidate profiles and anything behind a login carry personal data and contractual terms that a proxy does not change; handle them under your own data-protection obligations, or not at all.

Four copies, one opening

For example, suppose one warehouse supervisor opening is captured on October 5 from four sources. The rows might look like this (the employer and dates are illustrative):

SourceTitle as shownCompany as shownLocation as shownDate as shown
Employer careers pageWarehouse Shift SupervisorAcme LogisticsColumbus, OHPosted Oct 1
Job board 1Warehouse Supervisor (Nights)ACME Logistics Inc.Columbus, Ohio, United States4 days ago
Job board 2Shift Supervisor, WarehouseAcme LogisticsColumbus, OH 43215Posted today
Staffing agencyNight Warehouse SupervisorConfidential clientColumbus areaOct 3

How to collapse it:

  • Normalize the company. Strip legal suffixes and case: "ACME Logistics Inc." and "Acme Logistics" are the same employer. Keep the raw string too, for audit.
  • Normalize the location to city, region and country, plus a remote flag. "Columbus, OH 43215" and "Columbus, Ohio, United States" both become Columbus, OH, US.
  • Match titles loosely, not exactly. Token overlap after removing words like "shift" and "nights" is enough to propose a match; the company and location must agree before you accept it.
  • Convert relative dates using the capture time. "4 days ago" captured on October 5 means October 1. Store the board's time zone if it shows one.
  • Take the earliest credible date as posted. Job board 2's "Posted today" is a repost or a refresh of an opening first seen on October 1. Count it as activity on that opening, not as a new one.
  • Keep the agency copy separate. With the employer hidden, company matching is impossible. Link it as a possible match when the location and description text agree, but don't merge it without a shared requisition ID or a near-identical description.

The dedup key that falls out of this is normalized company, normalized title tokens and normalized location. Reports on "new openings this week" should count first-seen dates on that key, or reposts inflate demand.

Getting the right country's listings

Job boards localize. The same search for "warehouse supervisor" returns different results, and sometimes a different site edition, depending on where the request comes from. Check that the board's country edition and language match what you asked for before you store any results.

Node4's datacenter exits are on Node4-owned IP blocks in the United States, Italy and Spain, which covers boards in those three markets. For other countries, residential proxies use a partner network in 160+ countries, selected with a country segment in the username; the targeting guide has the syntax. Some boards respond differently to datacenter and household addresses, so run a sample search through each before you choose.

Paginated searches and sessions

A search result list is usually paginated, and some boards tie later pages to the first page's session. Fetch page four from a different exit and you can get page one again, which looks like duplicates in your data and hides the tail of the results.

Rotating gateways open a new exit for each HTTPS tunnel (and each plain-HTTP request); requests that share an open tunnel share its exit. To hold one exit across a whole search, include a session ID:

curl -x "http://USERNAME-country-us-session-job5518:PASSWORD@gw-rotating_datacenter.node4.io:8083" https://api.ipify.org

On Rotating Datacenter and Rotating Unmetered plans, a session ID pins one exit for up to 30 minutes from first assignment, and more traffic does not extend it. On residential plans, sessions are best effort, with a 30-minute ceiling; one can end sooner if the device behind the exit goes offline. Keep each search small enough to finish inside its session (one title in one city, say), and when the exit changes partway, rerun that search from the first page. Individual posting pages you already have URLs for can be fetched without a session. More detail is in sticky versus rotating sessions.

Schedule and cost

Postings change on a daily rhythm, so most agencies need one pass a day per search, plus a quick recheck of known openings to catch closures. A closed posting usually shows an explicit "no longer accepting applications" page; a refused request does not mean the job closed, so keep the opening open until a normal page says otherwise.

Measure the transfer of one day's pass before choosing a billing model; the proxy calculator turns requests per run and average transfer per request into the bandwidth one run needs. Rotating Datacenter is billed per GB (rotating prices). If you run very large search grids and would rather pay by concurrent connections with no per-GB charge, see connection-based pricing.

Where to start

Choose ten job titles in two cities. Collect them from the employer sites and two boards, run the dedup steps above, and check by hand that each real opening ended up as one row with the right first-seen date. The proxy providers for Python developers article covers the client libraries most collectors are built on. Property listings have the same repost problem; the real estate data guide handles it with listing IDs and update times.