How to Test a Machine Learning Model for Bias

Where bias actually shows up in business models

When people hear 'algorithmic bias' they picture facial recognition failures or national scandals. The everyday version is quieter. A model scoring sales leads that consistently ranks enquiries from certain postcodes lower. A churn model that flags older customers for retention offers less often. A CV screening tool that learned from ten years of hiring decisions made by the same three managers.

None of these models were built to discriminate. They learned patterns from historical data, and history contains the decisions people made. The model just makes them faster and with more apparent authority.

If a model influences decisions about people, it should be tested for bias before it goes live and periodically after. That is sound ethics, it reduces legal exposure under equality and consumer law, and in some uses it is becoming a regulatory expectation. New York City's rules on automated employment decision tools, for instance, require independent bias audits, and the EU AI Act's high-risk regime expects data governance that addresses bias.

Which fairness metrics to check

There is no single fairness number. Different metrics capture different ideas of fair, and it is mathematically impossible to satisfy all of them at once in most real situations. Pick the ones that match the harm you are worried about.

In most business settings we look at all four and focus the conversation on the one tied to the real-world harm. A lender worries most about wrongly declining creditworthy applicants from a particular group. A fraud team worries most about wrongly flagging honest customers.

The problem of missing demographic data

Here is the obstacle most businesses hit immediately. To measure outcomes by group, you need to know which group each person belongs to. Most companies, sensibly, do not collect ethnicity or religion in their CRM.

There are several workable options, each with trade-offs:

• Use characteristics you legitimately hold. Age and sex are often available. Test those properly even if others are not possible.
• Voluntary survey data. Equal opportunities monitoring in hiring is the classic example. Keep it separated from the model and use it only for testing.
• Statistical inference. Estimating group membership from names and locations for aggregate analysis. Useful for spotting large disparities, error-prone for individuals, and must never feed back into decisions.
• Synthetic test cases. Create matched pairs of records that differ only in a characteristic or proxy, and compare scores. Good for finding direct sensitivity.

Handling protected characteristic data for testing has its own data protection implications. Get advice on lawful basis before collecting or inferring it.

Proxies: the bias that survives removing the column

Removing sex, age or ethnicity from the training data does not make a model fair. Other variables often carry the same information. Postcode correlates with ethnicity and income. Career gaps correlate with sex. Years of experience correlate with age. Even the email domain someone uses can say something about their age.

To find proxies:

1. Check how well the remaining features predict the protected characteristic. If a simple model can guess it accurately, the information is still in there.
2. Look at feature importance for the main model and ask, for each strong feature, whether it could be standing in for something it should not be.
3. Run the matched-pair test above, varying only the suspected proxy.

Sometimes the proxy is also genuinely predictive for legitimate reasons, and removing it hurts the model for everyone. That is a judgement call for people, not something the data resolves.

A bias testing process that fits a normal ML project

1. Before building, write down which groups you will test, which metrics matter and what size of gap would trigger action. Deciding after seeing results invites rationalisation.
2. Check the training data for representation. A group that makes up 3% of training examples will usually get worse predictions.
3. Evaluate on a held-out test set broken down by group, with enough examples per group that differences are not just noise.
4. Investigate every gap above threshold. Is it the data, a proxy, or a real difference in the underlying outcome?
5. Mitigate by rebalancing data, removing or constraining proxy features, adjusting thresholds by group where lawful, or adding human review for affected cases.
6. Document what was tested, found and changed. A model card is the natural place.
7. Monitor after launch, because the population the model sees will drift.

This is built into how we run machine learning projects at SpiderHunts whenever a model touches decisions about individuals. It adds some days to a project, not months.

When a fair model is not achievable

Sometimes testing shows a gap you cannot close without making the model useless. The historical data may be too skewed, or the thing you are predicting may itself reflect past unfairness.

The honest options then are to narrow the model's role (a tool to prioritise human review rather than to decide), to keep humans making the decision with the model as one input among several, or to not use a model for that decision at all. We have recommended the last one more than once. A manual process that is slow but defensible is better than an automated one that quietly repeats the past.

A model trained on past decisions learns what you did, not what you should have done.

Bias in language model applications

Generative AI raises related but different issues. A model drafting job adverts, summarising candidate interviews or answering customer questions can treat groups differently in tone, detail or recommendations. The testing approach is similar in spirit: build sets of matched prompts that vary a name, a stated age or a location, and compare outputs systematically. Our guide to evaluating language models for business covers how to build that kind of test set.

Frequently asked questions