Biases and data problems that make AI fail (and how to avoid them)

An AI system is only as good as the data it uses. A huge share of AI project failures can be explained by data problems: data that is incomplete, duplicated, out of date or, worse, biased. This article reviews the most common problems, how to detect them and what to do about them.

What is a bias?

A bias is a systematic error that makes the system perform worse for, or harm, a group of people. It is not a one-off failure: it always repeats in the same direction.

The most common biases

• Historical: the data reflects past inequalities and the system reproduces them.
• Representation: some groups are under-represented in the data (older people, migrants, people with disabilities) and the system is less accurate for them.
• Proxy variables: apparently neutral data that stands in for a protected characteristic, such as neighbourhood, postcode or language.
• Automation: accepting what the AI proposes without reviewing it.
• Stereotypes in generated content: text and images with gender or cultural stereotypes.
• Selection: who ends up in the data is not random; for example, only people who complete the procedure online.
• Survivorship: you only see the cases that remained in the system (the applications that were resolved, not the abandoned ones), and precisely the cases that would explain the problem are missing.
• Measurement: the form or instrument measures differently depending on the group.
• Aggregation: the overall average hides, or even reverses, what happens in each group (Simpson’s paradox). No aggregate result is accepted without breaking it down.
• Temporal drift: reality changes while the data still describes the past.
• Deployment: using a pattern learned in another context or at another time to make decisions now.
• Confirmation: the analyst stops looking once they find what they expected.

Removing the sensitive data does not remove the bias

This is the most widespread mistake. If you remove sex or nationality, other variables reconstruct it: name, neighbourhood, school. The right approach is to keep the data in a controlled way, with a legal basis and protective measures, so that differences between groups can be measured, and to decide explicitly whether or not the system uses it.

Other data problems that lead to wrong results

• Disguised nulls and default values: zeros, -1, 999, dates such as 1900-01-01 or the option preselected in the form look like valid data and end up in the calculations. If the default values for education and experience are accepted when people are registered, the system learns from the default value, not from the person.
• Anomalies: extreme values, duplicates, mixed units or form caps. Not everything unusual is an error: in fraud or risk, the unusual is often what matters. Review before deleting.
• Single-source data: a system trained on data from a single site or period learns its peculiarities, not the phenomenon.
• Imbalanced classes: if a case is rare (fraud, for example), a system that always answers “no” is almost always right and detects none of them. You need to measure how many rare cases it detects, and by group.
• Data leakage in training: using data that will not exist at the moment of the decision, or that contains the answer. Spectacular test results and failure in production.

How to detect them: exploratory data analysis (EDA)

Before using a dataset, an exploratory analysis answers three questions:

1. Are all groups proportionally represented?
2. Are there variables acting as proxies for protected characteristics?
3. Do the system’s results differ between groups?

And when assessing each AI use case, three more: who could it harm? Does it work equally well for everyone? Is there a less intrusive alternative?

How to correct a detected bias

Depending on where you intervene:

Stage	Technique
In the data, before training	Rebalance or reweight the groups and collect more representative data
During training	Add a fairness constraint to the model’s objective
In the predictions	Adjust decision thresholds by group

First you have to decide which kind of fairness you are aiming for (the same proportion of favourable outcomes or the same accuracy rate in each group, for example), because there is no single definition and some are mutually incompatible. And if the bias is historical, there is no technical fix: it is a governance decision about what use the system is allowed for.

Turning each finding into a rule

Each problem found becomes a validation rule that is checked on every data load: schema, range, value catalogue, completeness, uniqueness, consistency between fields and stability of proportions against a baseline. Each rule has an owner and a severity:

Severity	Response
Blocking	The load is stopped and the data owner is notified
Critical	Only continues with explicit approval
Warning	Continues and is logged
Informational	Only logged

Tools: for profiling and exploration, pandas or ydata-profiling; to keep the rules written down, versioned and executable on every load, Great Expectations or Pandera; and to inventory sources, lineage and owners, a catalogue such as Microsoft Purview or OpenMetadata.

Three examples of how bias appears

Nursery school places. A system that prioritises applications is trained on historical data in which certain neighbourhoods had fewer accepted applications. Without prior analysis, it learns that the neighbourhood predicts acceptance and perpetuates the difference: indirect discrimination based on place of residence, even though the neighbourhood never appears as a criterion.

Job seeker register. If the default values for education and experience are accepted when people are registered, a large proportion of the records have the same value. A recommender trained on that data learns from the default value, not from the person, and recommends the same thing to everyone.

Fraud detection. If only 1 % of cases are fraud, a model that always says “no fraud” is 99 % accurate. Overall accuracy is useless: you need to measure how many frauds it detects (sensitivity) and how many false alarms it generates, and do so by group, because the system may detect well in one group and poorly in another.

How to measure whether a system is fair

Some common metrics, which should be calculated for each relevant group (age, sex, nationality, area, disability, language):

Metric	What it compares	When it matters
Favourable outcome rate	What proportion of each group receives the positive outcome	Grants, selections, prioritisation
True positive rate	Of the cases that really deserved it, how many it detects in each group	Detecting needs or risks
False positive rate	How many unjustified alarms each group receives	Checks, inspections, fraud
Accuracy by group	Whether the system is equally accurate for all	Any classifier
Calibration	Whether the same score means the same thing in every group	Scoring systems

Not all of them can be equalised at the same time: you have to decide which one matters in each case and document it.

Exploratory analysis checklist

• [ ] How many records are there per group, and are they proportional to the population the system will be applied to?
• [ ] What percentage of nulls, default values and disguised nulls does each field contain?
• [ ] Are there duplicates, mixed units or out-of-range values?
• [ ] Is any variable highly correlated with a protected characteristic?
• [ ] Where does the data come from? A single site, channel or period?
• [ ] Are abandoned, rejected or incomplete cases missing?
• [ ] Are there variables that will not exist at the moment of the decision or that contain the answer?
• [ ] Are the proportions stable over time or is there drift?
• [ ] Do the aggregate results hold when broken down by group?

Who does what

• The business owner decides which kind of fairness is sought and accepts the residual risks.
• The technical team carries out the exploratory analysis, measures by group and applies the corrections.
• Data protection validates the legal basis for keeping and using sensitive variables for measurement purposes.
• The body that authorises AI uses can reject a lawful system if its impact is not acceptable.

Monitoring during operation

Biases are not solved once and for all. During operation you need to review results by group periodically, re-run the test bench after every change of model or data, and watch for temporal drift.

At Bidatia we carry out data quality and bias analyses before every AI project, and we leave the rules automated. If you want to go further, see data quality: dimensions and metrics and the lifecycle of an AI project.