Check what you are permitted to do
Before building anything, read the site's terms and its robots file. Some sites permit collection, some prohibit it, and some offer an API that makes scraping unnecessary.
An API, where one exists, is always better: stable, documented, permitted and less work. Check for one before building a scraper.
Be a considerate client
- Rate limit yourself well below anything that would burden the site
- Identify yourself honestly in the user agent
- Cache so you do not fetch the same page repeatedly
- Respect the robots file
- Back off when you receive errors rather than retrying immediately
Structure changes will break it
Websites change without notice. A scraper depending on a specific structure will break, and the question is only whether you find out immediately or in six weeks.
- Validate that extracted data looks plausible before storing it
- Alert when the extraction rate drops
- Alert when a field is suddenly empty across all records
- Store the raw response so a change can be diagnosed
Monitor for silent failure
| Signal | Means |
|---|---|
| Zero records extracted | Structure changed or blocked |
| Records extracted but fields empty | Selectors broken |
| Values outside expected ranges | Parsing wrong |
| Sudden volume change | Source changed |
| Increasing error responses | Being rate limited or blocked |
A scraper that returns nothing and reports success is the standard failure, and it is entirely preventable with a plausibility check.
Handle personal data properly
Publicly accessible does not mean freely usable. Personal data collected from the web carries the same obligations as personal data collected any other way.
Decide the lawful basis, the retention and the purpose before collecting, rather than after.