The data you need, and a clause that says no
The project made sense: pull listings, prices or company data from a particular website into your system. Then someone read the terms of use and found a clause banning scrapers, robots and automated access. Or the developer you hired mentioned it, or the site blocked the first test run.
Now the business need is still there, but the obvious route looks off limits, and you are not sure how seriously to take it.
Why sites forbid automated access
Site owners restrict scraping for understandable reasons. The data may be their product, which they sell. Heavy automated traffic costs them money and can slow the site for real users. They may be protecting personal data about their users. And they may have commercial agreements with data partners that scraping undercuts.
Whether a particular terms clause is enforceable in a given situation is a legal question that depends on the facts, the jurisdiction and how the site is accessed. That is for a lawyer. The practical point is simpler: ignoring the terms puts you in a dispute you would rather not have, and the site owner controls the other end.
| Risk | What it looks like in practice |
|---|---|
| Contractual | A claim that you breached the terms you accepted, especially if you logged in |
| Data protection | Collecting personal data without a lawful basis under UK GDPR |
| Database and copyright rights | Copying a substantial part of a protected database |
| Technical | IP blocks, CAPTCHAs, legal letters to your hosting provider |
| Reputational | A partner or customer learning your data came from a scraper they had banned |
What pushing ahead anyway costs
A collector built against a hostile site is a maintenance burden. It gets blocked, gets rebuilt, gets blocked again. Data arrives with gaps. And if the site owner sends a letter, the project stops abruptly, often after you have built processes on top of it.
For businesses that sell to larger companies, there is also due diligence. Buyers and investors increasingly ask where your data comes from, and an honest answer of we scrape it against their terms does not go down well.
How we find a legitimate route to the data
- We look at exactly what data you need and why. Often it is a narrower slice than first assumed, which widens the options.
- We check for an official API, including partner or developer programmes that are not linked from the homepage.
- We check whether the site owner sells the data under licence, and help you compare that cost with the cost of an unreliable scraper.
- We look for other sources of the same facts: open government data, Companies House, Land Registry, official registers, or a different provider with permissive terms.
- We draft a request to the site owner for permission or a feed. Plenty of site owners agree to a limited, polite arrangement when asked.
- If none of those work and the volume is small, we build a tool that makes a person's manual lookup faster, such as a structured form beside the site, rather than an automated collector.
- Whatever route is chosen, we build the integration into your system with validation, logging and a record of where each field came from.
We do not build collectors designed to evade blocks on sites that forbid them. That is a line we hold because it protects you as much as us.
Data you can explain to anyone
You end up with a source you can name without hesitation, a route that does not break every time the site notices, and a data pipeline built on that route. If the licence has a cost, you know what it is and what it buys. If the answer is open data or a permitted API, the build is usually simpler and more stable than scraping would have been.
And if the data really cannot be obtained legitimately, you find that out before investing in a system that depends on it.
Are you in this position?
- The website you need data from prohibits scraping in its terms
- A developer has warned you about the site's terms
- Your first scraping attempts were blocked
- You are unsure whether to ask the site owner for access
- Customers or investors may ask where your data comes from