403, 429 and a CAPTCHA
The collector was working. Then the responses changed: error 403, error 429, a page asking you to prove you are human, or a normal-looking page with no data in it. Someone suggested rotating proxies and fake browser fingerprints, and now you are wondering whether that is a good idea.
Two very different reasons for a block
Before changing anything, it matters which situation you are in, because the right response is opposite in each case.
| Situation | Signs | Right response |
|---|---|---|
| Your collector is too aggressive | 429 rate-limit errors, blocks after bursts | Slow down, cache, request less |
| The site does not want automated access | Terms forbid it, bot protection on every page | Find an approved route instead |
| The collector looks broken or suspicious | Missing headers, odd patterns, errors | Behave like a well-mannered client |
Many blocks are the first kind. A script that fires hundreds of requests a minute, re-downloads pages that have not changed, and runs at the same second every hour looks like an attack to any site's protection.
The second kind is different. If a site has put bot protection in front of its content and its terms forbid scraping, it has told you clearly what it wants. Rotating proxies and faking browsers to defeat that is evasion, it escalates legal and reputational risk, and it turns maintenance into an arms race you will not win.
What the blocking costs
Gaps in data exactly when you need it. Developer time spent on workarounds. Proxy services that cost money every month. And if you evade a clear refusal, the risk of a formal complaint or letter, which can end the project and damage relationships with people you may want to do business with.
How we deal with blocks properly
- We read the site's terms and robots.txt and look at the responses you are getting, to work out which situation you are in.
- If the collector is simply too aggressive, we slow it down, spread requests over time, add backoff when the site signals it is busy, and stop requesting pages that have not changed by using conditional requests and caching.
- We reduce the number of requests needed, for example by using the site's sitemap or listing pages instead of crawling every link, or its structured data feed where one exists.
- The collector identifies itself honestly with a user agent that includes contact details, where appropriate.
- If the site does not want automated access, we stop and look for an alternative: an official API, a paid data licence, a partner feed, open data with the same facts, or a written request for permission.
- Whatever route is used, the collection is monitored, and any new blocks or errors alert a named person rather than silently producing gaps.
We do not solve CAPTCHAs automatically, use residential proxy networks to hide the source, or fake identities to get past protection a site has deliberately put in place.
Collection that is not a fight
Where the site tolerates polite collection, the collector runs quietly with few errors and no drama. Where it does not, you have an approved source that does not need defending. Either way the data arrives steadily, and you can explain where it came from to a customer, investor or regulator.
You also stop paying for workarounds that only postponed the next block.
It is worth remembering that a site blocking you is information. It tells you the owner cares about how their data is used. Treating that as a conversation rather than an obstacle is often how a better arrangement, such as a data feed, begins.
Is this what you are seeing?
- Your collector gets 403 or 429 errors
- CAPTCHAs appear after the collector runs
- Someone has suggested proxies to get around blocks
- You are not sure whether the site allows collection at all