Is Web Scraping Legal? A Risk Screen for Engineers

Where US and EU courts draw the line on scraping, the five risks to screen a project against, and the controls that keep a crawler defensible.

Yes, in most of the situations an engineer is likely to be in, and no in a few specific ones that are easy to name. Scraping data that a site serves to anyone, without logging in and without defeating a technical barrier, has not been treated as a computer-crime offense by the US courts that have examined it, and no general statute forbids it in the EU or the UK. What turns a lawful crawl into a claim is one of five things: getting past an access control, breaching terms you agreed to, reproducing protected content, processing personal data without a basis, or hitting a server hard enough to harm it. This guide is not legal advice. It is the risk screen an engineering team can run in ten minutes before a line of code is written, with the case law that shapes each question, including the places where "probably fine" is the most anyone can say.

The short answer, by jurisdiction

United States. The statute people fear is the Computer Fraud and Abuse Act, and two decisions narrowed it. In Van Buren v. United States (2021) the Supreme Court read "exceeds authorized access" as a gates-up-or-down question: either you were allowed into the system or you were not, and misusing information you were entitled to reach does not "exceed authorized access" under that statute. On remand in hiQ Labs v. LinkedIn (2022), the Ninth Circuit applied that reading to the public web. It was ruling on a preliminary injunction, not on the merits, and what it said was that when a site permits public access to its data, scraping that data is likely not access "without authorization" under the CFAA, even after a cease-and-desist letter. Note how hiQ ended, though. The case settled with hiQ agreeing to stop and to delete the data, on LinkedIn's contract and account-misuse claims, not the CFAA one. The criminal exposure narrowed; the contract exposure did not.

The contract question then got a decision of its own. In Meta v. Bright Data (2024) a federal court granted summary judgment to Bright Data on Meta's breach-of-contract claim, because the scraping at issue was of public, logged-out pages, and the terms Meta relied on governed the conduct of logged-in users. The line the courts keep drawing is not "public versus private data". It is "did you accept terms, and did you go around a gate".

European Union. There is no scraping statute. Three regimes apply instead. The GDPR governs any personal data you collect, and Article 3 reaches you when you monitor the behavior of people in the Union regardless of where your servers sit. The Database Directive gives the maker of a database a sui generis right against extraction of a substantial part of it, which has no US equivalent and matters enormously for a catalog or a listings site. And the 2019 copyright directive created two text-and-data-mining exceptions for content you have lawful access to. Article 3 covers scientific research by research organizations and cannot be switched off. Article 4 covers everyone else, and a rights-holder can switch it off by reserving the right in an appropriate manner, which for content published online means machine-readable means such as metadata or the site's terms. In practice that reservation is expressed in robots.txt, a dedicated TDM reservation file, or the terms page, and it is the reason the crawl-policy check below is a legal input rather than a courtesy.

United Kingdom. Broadly the EU position, carried over: UK GDPR, a retained database right, and the Computer Misuse Act. Its section 1 offense is knowingly causing a computer to perform a function to secure access you are not authorized to have. The statute has no technical-barrier element, so how far it reaches pages a site serves to anyone is less settled than the US position, and a cease-and-desist letter is the point at which to stop and ask.

Everywhere, the short answer has the same shape. Public, logged out, rate limited, fields rather than pages, personal data minimized: low risk. Behind a login you created for the purpose, past a CAPTCHA, ignoring a reservation, or copying a whole database: a claim waiting for a plaintiff with a budget.

Ethical and legal are two different screens

"Ethical" and "legal" are not the same word, and treating them as synonyms gets people into trouble. Legal web scraping is the narrower question: did you break a specific law or an enforceable contract? Ethical scraping is broader. It asks whether you respected the site owner's reasonable expectations, the people whose data appears on the page, and the server footing the bill for your traffic.

The biggest misconception is that public availability equals permission. A page loading in a browser without a login does not mean the operator consented to bulk automated extraction, and it says nothing about whether the content includes personal data protected under the GDPR or a similar regime. The decisions above distinguish sharply between accessing public pages and defeating a login wall or a blocklist, and a project can be lawful on the first count and still be the reason a small site's hosting bill doubled.

The practical fix is a risk-based approach. For every project, ask what type of data you are collecting, how you are accessing it, and why. A price-comparison crawl on public product pages carries a different risk profile from scraping user profiles behind a free account.

The five risk axes

Five distinct legal questions apply to almost any scraping project, and they overlap less than people assume. A project can be low risk on one axis and high on another, so run each one separately rather than making a single gut call.

Authentication and technical gates. If you had to log in, defeat a CAPTCHA, or work around a block to reach the data, treat this as high risk. Scraping logged-out public pages is unlikely to be a CFAA violation on the current case law; circumventing a barrier changes that calculation fast. Mitigation: stay logged out, and never reuse credentials or tokens that are not yours.
Contract and terms of service. If the site's terms explicitly prohibit automated access and you accepted them, by creating an account or by clicking through, you have breach-of-contract exposure even where no statute applies. Terms you never assented to are weaker, which is the whole of the Bright Data result. Mitigation: read the terms before you build anything, and document your reasoning if you proceed.
Copyright and database rights. If you are reproducing substantial creative content or, in the EU, extracting a substantial part of a protected database, you risk an infringement claim. Mitigation: extract discrete facts and fields rather than wholesale page copies, and check jurisdiction, because the sui generis database right has no direct US equivalent.
Personal data. If the data includes names, contact details, or anything identifying a real person, the GDPR's transparency, purpose-limitation and data minimization requirements can reach you even when your servers sit outside the EU. Mitigation: strip fields you do not need before storage, not after.
Server harm. If your crawl pattern could plausibly be mistaken for a denial-of-service attempt, you are in dangerous territory on both legal and reputational grounds. Public advisories such as the FBI's on home devices conscripted into criminal proxy networks are a reminder of why security teams treat unexplained automated traffic as hostile until it proves otherwise, and an unthrottled crawl is exactly the traffic that invites that reading.

Jurisdiction changes the math on every axis. EU projects inherit database rights and the GDPR's extraterritorial reach that US-only projects do not face. When a project touches sensitive categories, cross-border personal data, or anything resembling research on human subjects, involve counsel before you write a line of code.

The compliance checklist

Most scraping disputes trace back to a handful of avoidable decisions made early. This list orders the controls by how much risk each one removes, starting with the highest-leverage move.

Check for an official API or licensed feed first. If the target offers one, use it. Going around a paywall or a login to avoid a fee is the fastest way to convert a gray-area project into a clear violation.
Read robots.txt and any TDM reservation before crawling. Document what you found and how your crawler honors it. The robots.txt tester will tell you what a given path resolves to under a site's rules before you commit a crawler to it.
Rate-limit hard, and run off-peak where feasible. A crawler indistinguishable from a person clicking slowly is far less likely to trigger abuse detection or server strain.
Collect only the fields the project needs. If you do not need email addresses or user IDs, do not extract them, even when they are sitting in the HTML.
Exclude sensitive personal-data categories outright unless you have a specific, documented lawful basis. Health, political opinion, and similar categories deserve automatic filtering.
Keep provenance logs: timestamp, source URL, exit address, user agent, retention period, for every batch. This is what a regulator or opposing counsel asks for first, so build the log as a first-class output of the scraper rather than something bolted on later; a clean audit trail is worth more than any argument constructed after the fact.
Set a deletion timeline and keep to it. Data you will not need in six months should not still be in a warehouse then.
Put high-risk projects through internal review before they start, not after a complaint arrives. Anything touching personal data at scale, competitor pricing, or vulnerable populations deserves a second set of eyes.

The CNIL's guidance on legitimate interest makes the same point from the regulator's side: legitimate interest can justify scraping personal data, but only where you can show documented safeguards, such as automatic exclusion of irrelevant sensitive fields.

Engineering controls that turn policy into practice

Policy documents do not crawl websites. Code does, and the parameters you choose decide whether a project stays defensible or turns into an incident report.

Start with concurrency and pacing. Keep concurrent requests to a single site in the single digits, and use randomized delays rather than a fixed interval, since uniform timing is itself a signal that flags automated traffic. Slowing down when observed response latency climbs is one of the operational controls set out in an individual Internet-Draft on automated public data collection submitted to the IETF. Internet-Drafts are working documents rather than adopted standards, so treat it as a well-argued reference, not a rule anyone can hold you to.

Rotation belongs in the same conversation, because the point of it is to keep per-address request rates low rather than to hide: the mechanics, and what a realistic ban rate actually looks like, are in rotating proxies for web scraping.

Treat robots.txt as a programmatic input, not a suggestion you skim once. Parse it at run time, and if you decide a specific exception is justified (some crawl-delay directives are set unreasonably high), write down why before you override it.

Session handling deserves its own rule: stay logged out wherever the project allows it. An authenticated session raises the stakes on every axis above, and a scraper using someone else's credentials or a scraped token is functioning closer to unauthorized access than to data collection.

Build retry logic around server signals, not stubbornness:

On a 429 or a 503, back off immediately and exponentially, not with a quick retry.
Cap total concurrency per domain regardless of how many workers your infrastructure could run.
Log every request's metadata for later audit, including the failures.
Pseudonymize or drop personal identifiers at the point of extraction rather than after storage.

A crawler that has never once received a 429 is either scraping a very quiet corner of the internet or running with headroom it does not need; an occasional throttle is the site's rate limiter and yours agreeing on a pace. The operational half of this, budgets with jitter, conditional requests, resumable jobs and catching the 200s that quietly contain nothing, is in web scraping best practices.

The ten-minute screen

Before any code ships, run the project through this pass. It catches most of the expensive mistakes.

Does the target offer an API or a licensed feed that covers the use case? If yes, use it instead.
Would collecting this data require getting past a login, a CAPTCHA, or a paywall? If yes, stop and reconsider the approach.
Does the dataset include personal data, and if so, is there a documented lawful basis and a minimization plan?
Do the site's terms explicitly prohibit automated access, did you assent to them, and how much contractual risk is the business willing to accept?
Could the planned request volume plausibly be mistaken for a denial-of-service pattern by the target's security team?
Does the project touch vulnerable populations, sensitive categories, or research on human subjects, and would an ethics review make sense first?

If two or more answers point toward risk, the minimum bar for proceeding is a documented lawful basis, sign-off from someone outside the engineering team, and logged technical safeguards: rate limits, minimization, provenance. If the honest answer to several questions is "we are not sure", that uncertainty is itself the signal to consult counsel or to pivot to a licensed source, a manual sample, or a partnership with the data owner instead of scraping around the problem.

Where proxies fit, and where they do not

A proxy does not change any answer above. It changes which address the target sees, and that is all. There is one place a proxy can make the legal picture worse rather than better: rotating around a block. In Craigslist v. 3Taps (2013) a court held that once the site had blocked the scraper's addresses and sent a cease-and-desist letter, continuing through fresh addresses was access "without authorization" under the CFAA. The Ninth Circuit's later reasoning in hiQ, that data a site serves to the public without an authorization system is likely not gated for CFAA purposes, cuts against that theory, but hiQ did not address 3Taps, which remains a district-court ruling nobody has overturned, and the fact pattern is still the one that turns a scraping dispute into a computer-misuse allegation. If a site has blocked you and told you to stop, the question is no longer engineering.

Short of that, a proxy is infrastructure for the checklist. Owned address blocks, which is what Node4 operates for its datacenter and rotating products, keep traffic patterns consistent and attributable; residential coverage, which Node4 sources from a partner network rather than owning, is for the targets that reject hosting ranges outright. Usage reporting shows request volume and bytes per proxy, while the response codes your targets return stay inside the tunnel, so that half of the provenance log is yours to write. Country-level exits also matter for one reason: a price or a search result is only evidence of what a person in that country would see if it was collected from that country. Price monitoring and rank tracking both turn on that.

Before choosing a proxy type at all, do I need proxies is the shorter question, and the use-cases page maps the common projects onto the rate-limiting and provenance principles above. A provider checklist covers what to ask before you pay anyone, including us.