The HTML is there, the prices are not
Someone on your team, or a freelancer, wrote a short Python script: fetch the page with requests, parse it with BeautifulSoup, find the product names and prices. It works on one site and returns nothing on another. Printing the HTML shows a skeleton page, some script tags and a loading message.
A quick search turns up the usual advice: use Selenium. It works, slowly, and now the script launches a whole browser for every page and falls over when run on a server.
Why BeautifulSoup sees nothing
BeautifulSoup is a parser. It reads whatever HTML it is given. It does not run JavaScript. Many modern sites send a nearly empty page and then use JavaScript to fetch the actual data from an API and draw it on screen. A browser shows you the finished result; requests only gets the empty starting page.
| Tool | What it does | Best suited to |
|---|---|---|
| requests with BeautifulSoup | Fetches and parses raw HTML | Pages where data is in the HTML |
| Scrapy | Crawling framework with scheduling, retries and pipelines | Many plain pages at volume |
| Selenium | Drives a real browser | Existing test setups, legacy scripts |
| Playwright | Drives a real browser, modern API, good waiting logic | Sites that genuinely need JavaScript |
| Direct API or JSON call | Requests the same data the page loads | Most JavaScript-driven sites, when permitted |
So the question is not really BeautifulSoup versus Selenium versus Playwright. It is where the data comes from, and what is the lightest acceptable way to get it.
What the wrong tool costs
A full browser for every page is slow and uses a lot of memory, so jobs that should be quick take far longer and cost more to host. Browser scripts are more fragile, breaking on pop-ups, cookie banners and timing. And running many browsers against a site produces heavy traffic that is more likely to be noticed and blocked.
On the other side, using plain requests where a browser is needed produces the silent empty results you started with, which is worse than an error because it can look like the site simply had no data today.
How we pick the right approach
- We check the site's terms and robots.txt, and whether it offers an official API or feed. If it does, that is the route.
- We open the browser's network tools and watch what the page loads. Often the data arrives as JSON from an endpoint, or is embedded in the page as a JSON block or structured data.
- Where reading that data is consistent with the site's terms, we request it directly with plain HTTP. It is faster, lighter and more stable than parsing the visual page.
- Where the data really only exists after JavaScript runs, we use Playwright in headless mode, waiting for specific elements rather than fixed delays, and reuse one browser across many pages.
- For many pages, we wrap the approach in Scrapy or a simple job queue with rate limits, retries and logging.
- Every record is validated so an empty or partial result is caught and reported, not stored as if it were real.
We do not use browsers to get around logins, paywalls or bot protection on sites that do not want to be collected.
A collector that returns what it should
The script returns data again, and when it does not, it says so. It runs on a server without a pile of browser processes, finishes in a sensible time and is less likely to upset the site. When someone later asks why it uses one tool on one site and another elsewhere, the reason is written down.
Does your scraper do this?
- It returns empty results on some sites but not others
- The fetched HTML contains a loading message or script tags only
- Someone switched to Selenium and it is now slow and flaky
- It struggles when run on a server rather than a laptop
- Empty results are stored as if they were real