Scraping That Keeps Working

Check what you are permitted to do

Before building anything, read the site's terms and its robots file. Some sites permit collection, some prohibit it, and some offer an API that makes scraping unnecessary.

An API, where one exists, is always better: stable, documented, permitted and less work. Check for one before building a scraper.

Be a considerate client

1. Rate limit yourself well below anything that would burden the site
2. Identify yourself honestly in the user agent
3. Cache so you do not fetch the same page repeatedly
4. Respect the robots file
5. Back off when you receive errors rather than retrying immediately

Structure changes will break it

Websites change without notice. A scraper depending on a specific structure will break, and the question is only whether you find out immediately or in six weeks.

• Validate that extracted data looks plausible before storing it
• Alert when the extraction rate drops
• Alert when a field is suddenly empty across all records
• Store the raw response so a change can be diagnosed

Monitor for silent failure

A scraper that returns nothing and reports success is the standard failure, and it is entirely preventable with a plausibility check.

Handle personal data properly

Publicly accessible does not mean freely usable. Personal data collected from the web carries the same obligations as personal data collected any other way.

Decide the lawful basis, the retention and the purpose before collecting, rather than after.

Frequently asked questions