Web Scraping, Done Properly

What businesses actually use it for

• Competitor pricing and availability monitoring
• Aggregating supplier catalogues that arrive as web pages
• Market and product research at a scale nobody can do by hand
• Qualifying leads from public company information
• Monitoring your own products across resellers and marketplaces

All of these are ordinary commercial activities. The controversy attaches to how it is done rather than to whether data is collected at all.

The boundaries we work within

1. Public data only. Nothing behind a login, nothing requiring a credential.
2. Terms of service matter. We check them, and we tell you when they prohibit what you are asking for.
3. Personal data is regulated. Public does not mean unregulated; GDPR applies to personal data wherever it was found.
4. Rate limiting, always. We collect politely; a scraper that degrades someone's site is both rude and a liability.
5. Use an API when one exists. It is more reliable, cheaper to maintain and unambiguous.

We turn down scraping work regularly — behind logins, personal data at scale, or anything the target's terms plainly forbid. It is not worth the exposure for either side.

The maintenance is the project

Building a scraper against a site today is often two or three days. Keeping it working for two years is the actual engagement, because sites change layout without notice and a scraper that breaks silently is worse than none.

• Structural validation — alert when the expected elements vanish
• Volume checks — a page returning three results instead of sixty is a failure
• Value sanity checks — a price of £0.01 or £900,000 is not a price
• Alerting to a person, not to a log file nobody reads

Designing for change

We separate fetching, parsing and storing so a layout change touches one small module. Selectors live in configuration rather than in code, so an adjustment is a config change rather than a deployment.

We also store the raw fetched page for a rolling window. When something looks wrong three days later, you can see exactly what was on the page rather than reasoning from the parsed output.

What it costs

The monthly figure covers infrastructure, proxies where needed, monitoring and the fixes when sites change. Anyone quoting a build with no ongoing cost has not maintained one.

When not to scrape

If an API exists, use it. If the data is available for purchase at a sensible price, buy it — it will be cleaner and it comes with a support contract. If the terms of service forbid it and the relationship matters, do not.

And if the data would only be useful at a scale that would burden the source, that is usually a sign the answer is a commercial arrangement rather than a technical one.

Frequently asked questions