Using Public and Third-Party Data to Improve Your Models

Your data knows your customers, not the world around them

A garden centre chain's sales data can tell you that the first warm weekend of spring is huge. It cannot tell you when next year's first warm weekend will be. A B2B supplier's CRM knows which customers stopped ordering, but not that half of them are in a sector that has just had a bad year.

External data fills those gaps. Used well, it lets a model explain variation that looked random from inside the business. Used badly, it adds cost, fragility and features that make no measurable difference. The trick is testing before committing.

Sources that often earn their place

• Calendar data. Bank holidays, school terms, religious festivals, major sporting events. Free, stable and frequently the single most useful addition to a demand model.
• Weather. Historical observations and forecasts from national meteorological services or commercial weather APIs. Powerful for retail, hospitality, energy, construction and field services.
• Official statistics. ONS and Census data in the UK, Eurostat in Europe, and equivalent national sources. Useful for area-level demographics and economic conditions.
• Company records. Companies House filings, sector codes, incorporation dates and filing history. Helpful in credit risk and B2B lead scoring.
• Geographic data. Postcode-level deprivation indices, distances to competitors, drive times. Good for site selection and local demand.
• Market and commodity prices. Fuel, energy, raw materials and exchange rates, for pricing and cost forecasting.
• Competitor and marketplace data. Public prices and listings collected responsibly, for pricing models.

Prove it helps before you pay for it

The test is simple and should be non-negotiable. Train the model on your own data alone and record performance on a held-out period. Add the external source, retrain on exactly the same rows, and compare. If the improvement is small or disappears when you test on a different period, the source is not worth its upkeep.

Many commercial data providers offer historical samples for evaluation. Ask for one covering at least a year that overlaps with your own history. A weather API subscription that improves a restaurant group's staffing forecast by a meaningful margin pays for itself quickly; one that shifts the numbers by a fraction does not.

A feature that sounds relevant is a hypothesis. Only a with-and-without test on your own data turns it into a reason to pay.

The prediction-time trap

This catches out more external data projects than anything else. When you train a model on history, you have the actual weather for every past day. When you make a real prediction for next Saturday, you only have a forecast. Train on actual weather and deploy with forecasts, and the model will be less accurate than testing suggested.

The same applies to official statistics published months after the period they describe, and to company filings that appear a year late. For each source, record when the data becomes available, not just the period it covers, and build training data using only what would have been published by then. We cover this wider problem in data leakage in machine learning.

A quick comparison of common sources

Licensing, privacy and scraping

Read the licence. Some free datasets permit commercial use with attribution; some forbid it. Commercial data licences may allow use in reports but not for training models, or may require you to delete derived data when the contract ends. That last clause can quietly make a trained model non-compliant.

Bought-in data about individuals raises data protection questions. You need a lawful basis, people need to have been told their data could be used this way, and you should be able to explain the source. For collecting public web data, our guide to web scraping for business data done properly covers the legal and practical points.

External feeds break, so plan for it

An API changes its format, a government site moves a file, a vendor goes out of business. When a model depends on external data, its pipeline needs checks that notice missing or late feeds, and a defined fallback, such as using the previous forecast or a version of the model trained without that source.

When SpiderHunts adds external data to a model, we build the with-and-without comparison into the project from the start and keep the fallback model ready. It is also worth checking whether a simpler source does the job: bank holidays and a basic weather feed often capture most of the value in demand forecasting. If you would like help identifying and testing sources, that sits within our data science service.

Frequently asked questions