An overnight job that is still running at lunch
The scraper started small and worked. Then the list of pages grew. Now it runs overnight, is still going when people arrive, and sometimes crashes halfway with a browser error, leaving a half-finished spreadsheet. Someone restarts it from the beginning.
Running it on a bigger machine helped briefly. Running several copies at once made the target site start refusing connections.
Where the time actually goes
Selenium drives a real browser. For each page it loads the HTML, the stylesheets, the scripts, the fonts, the images and the tracking tags, runs all the JavaScript, and only then lets your code read the page. Most of that has nothing to do with the data you want.
Selenium scripts also tend to wait with fixed sleeps, process pages strictly one after another, and restart the browser when anything goes wrong.
| Symptom | Common cause |
|---|---|
| Very slow per page | Loading every asset in a full browser |
| Slow overall | One page at a time, fixed sleeps |
| Crashes midway | Browser memory growth, unhandled pop-ups |
| Restarts from scratch | No record of which pages are done |
| Site starts blocking | Parallel browsers generating bursts of traffic |
Scrapy works differently. It requests pages over HTTP without rendering them, handles many requests concurrently within limits you set, retries failures and keeps track of what is done. When the data is in the HTML or an API response, that is a far better fit.
What a slow scraper costs
Data arrives late or incomplete, so reports built on it are unreliable. Hosting costs rise because browsers need a lot of memory. Staff time goes on babysitting and restarting. And the bursts of browser traffic make the site more likely to block you, which is the one outcome that stops everything.
The restarts deserve their own mention. A job that has to begin again from the first page after every crash repeats hours of work and sends the same requests to the site twice, which is wasteful for you and unwelcome for them.
How we speed up collection
- We confirm the site's terms and robots.txt permit what you are doing, and check for an API or feed first.
- We profile the current job: how many pages, how long each takes, and where the data actually comes from.
- Pages whose data is in the HTML or in a JSON response are moved to Scrapy or plain asynchronous HTTP requests.
- Pages that truly need JavaScript stay in a browser, usually Playwright, with images and unneeded assets blocked, smart waits instead of fixed sleeps, and one browser reused across many pages.
- We add polite concurrency limits and delays so the job is faster for you without hammering the site.
- Progress is saved as pages complete, so a failure resumes where it stopped rather than starting again.
- Results are validated and loaded into a database, with a run report and alerts if counts drop or errors rise.
Faster does not mean more aggressive. Most of the gain comes from not doing unnecessary work, not from hitting the site harder.
A job that finishes before anyone notices
The collection runs in a sensible window, finishes, and reports what it collected. When it fails, it picks up from where it stopped. It uses a modest server instead of a large one. And it is gentler on the target site, which makes blocking less likely.
The code is also easier to maintain, because site-specific parsing is separated from the crawling machinery.
Hosting becomes simpler too. A crawling job that no longer needs dozens of browser instances can run on a small server or a scheduled cloud function, and the saved progress means an interrupted run costs you minutes of repeat work rather than the whole night.
Recognise this?
- Your scraping job runs for hours
- It crashes and restarts from the beginning
- It uses Selenium for every page
- Adding parallel copies led to blocks
- Hosting it needs a large server