It worked last month
A developer or freelancer wrote a Python script to pull data from a website. It ran on a schedule, fed a spreadsheet, and everyone forgot about it. Then someone noticed the spreadsheet had not changed in two weeks, or worse, that it was full of blank prices.
The site had changed its layout. The script was patched. A month later it broke again, a different way. Now the person who wrote it has moved on and nobody wants to touch it.
Why scrapers are fragile
A scraper reads a web page that was designed for people, not programs. Anything the site owner changes, whether a redesign, a new class name or a cookie banner, can break it.
| Fragile pattern | More stable alternative |
|---|---|
| Long CSS or XPath selectors copied from the browser | Short selectors on meaningful attributes or labels |
| Reading visible HTML | Reading the JSON data the page loads, or structured data embedded in it |
| Full browser for every page | Plain HTTP where the data is in the response |
| No checks on output | Validation of every record |
| Silent failures | Health checks and alerts |
The bigger problem is usually not the break itself. It is that nobody knows it has happened, so bad or missing data flows into decisions.
What a fragile scraper costs
Repeated developer time on emergency fixes. Gaps in data history that cannot be recovered afterwards. Decisions made on blank or stale figures. And dependence on one person who understands the script.
The gaps deserve a second look. Unlike an internal system, a website does not keep yesterday's prices for you. If the scraper was down for a fortnight, that fortnight of competitor or market data is gone for good, and any trend analysis carries a hole in it.
There is a quieter risk as well. A badly behaved scraper that hammers a site with requests is more likely to be blocked, and it reflects on your business if the site owner traces it back to you.
How we rebuild scrapers to last
- We check whether the site offers an API, a feed, a sitemap or a data download. If it does, we use that instead.
- We look at how the page gets its data. Many modern sites load JSON from an internal endpoint or embed structured data, which changes far less often than the visible layout. Where using it fits the site's terms, we read that.
- Where HTML parsing is needed, we use short, meaningful selectors with fallbacks, in a framework like Scrapy for plain pages or Playwright for JavaScript-heavy ones.
- Every record is validated: required fields present, prices numeric and in range, counts within expected bounds compared with the last run.
- Each run reports its health. If validation fails or counts drop sharply, a named person is alerted and bad data is held back instead of being loaded.
- Site-specific parsing is isolated in small modules with saved sample pages as tests, so a fix touches one file and can be checked before release.
- Collection stays polite: sensible request rates, respect for robots.txt, and no bypassing of access controls.
A scraper you stop thinking about
Most of the time it just runs. When a site changes, you hear about it straight away from an alert, not weeks later from a blank report. The fix is small because the code is organised and tested, and it does not need the original author.
Because each run's output is validated, you also get a record of data quality over time. If a site starts showing fewer products, or prices start arriving without VAT, the change is visible in the run reports rather than hidden inside a spreadsheet nobody audits.
History stays complete because failures are caught before the gap grows.
Is this your scraper?
- It has broken more than once after a site change
- Failures were noticed by people, not by alerts
- Only one person understands the code
- It sometimes produces blank or wrong values
- It runs a full browser even for simple pages