Web Scraping: What Is Sensible, and What Is Asking For Trouble

The short version

There is no single law called “scraping law”, and this is not legal advice. What exists is a combination of contract terms, copyright, database rights, computer misuse provisions and data protection, and which of them apply depends entirely on what you collect and how.

In practice, the projects that cause trouble share recognisable features. The ones that do not are boring: public data, modest rate, no personal information, no circumvention of access controls.

Four questions that determine your risk

1. Is it personal data? Names, emails, profiles about identifiable people bring data protection obligations regardless of whether the information is public.
2. Did you agree to terms? Creating an account and accepting terms is a contract. Scraping past a login you agreed not to scrape is a different position from fetching an open page.
3. Are you circumventing access controls? Defeating authentication or paywalls moves the conversation somewhere much less comfortable.
4. What load are you placing on them? Aggressive rates that degrade someone's service are both rude and legally unhelpful.

The practices that keep projects clean

• Respect robots.txt as a default position, and document any deliberate exception with a reason
• Rate limit conservatively — a request every few seconds is usually plenty
• Identify your crawler honestly in the user agent with a contact address
• Cache aggressively so you never fetch the same page twice unnecessarily
• Collect only the fields you need, and delete what you no longer use
• Prefer an official API where one exists, even at a cost — it is cheaper than the alternative

The user-agent-with-contact-address point is worth more than it sounds. Site owners who can reach you will usually email before they block you, and most disputes end in a conversation.

What we decline

Bypassing authentication or paywalls. Harvesting personal contact details for unsolicited outreach. Anything designed to evade detection specifically because the operator has objected. Scraping at volumes that would degrade a small site.

This is not squeamishness — those projects have a habit of ending badly for the client, and the client tends to remember who built the thing.

The engineering realities nobody quotes for

Scraping is not a one-off build. Sites change layouts, add protections, and vary regionally. A pipeline that worked in March needs attention in June, and a project priced as a one-off delivery is priced wrongly.

Build for change, because change is guaranteed

Assume every source will break. Store the raw response alongside the extracted data so you can re-parse historical pages when you fix a selector. Alert on suspicious success — a scraper returning zero rows without erroring is the failure mode that goes unnoticed for weeks.

Monitor field-level completeness rather than just job status. A run that finishes successfully with prices missing from 80% of records is a failure that looks like a success on a dashboard.

Frequently asked questions