Your AI's 92% confidence probably measures nothing

An apparently innocent number, somewhere between 85 and 95, moving just enough to look alive. Clinicians read it. Some of them weighted their decisions with it.

It was a lie. Not because anyone faked it, but in a worse sense: nobody had ever checked that it measured anything. When I finally audited it, that 92% did not come from the machine learning model we believed it came from. It came from a formula adding a fixed value to a questionnaire completeness ratio. It was saturated: past a certain point, a nearly empty case and a fully worked one produced practically the same number.

Why this happens so often

Because showing a percentage is trivial and calibrating one is hard, and in between there is a comfortable trap: the number looks good in the interface, the client asks for it, and since nobody knows what the correct value would be, nobody notices that the one on screen is not it.

The usual origins of a percentage that means nothing:

• A heuristic dressed as a model. A formula someone wrote in an afternoon that was never replaced, and that everyone now assumes comes from the model.
• A raw model probability. Neural networks are famously overconfident. Without calibration, that output is not a probability of being right, it is just a number between zero and one.
• The completeness of the input. The more fields filled in, the higher the score. It measures how much the user typed, not how likely the answer is to be correct.
• A model that exists but never runs. The trickiest one, because the system does have a correct component that never gets invoked. The interface may even name the model it used, and that label is lying too.

How to check yours this afternoon

You do not need data science. Three tests are enough:

• The range test. Run a hundred real cases and plot the distribution. If everything lands in a narrow band, the number is not discriminating anything.
• The absurd case test. Feed it something incoherent, or an input the system should not be able to answer. If confidence stays high, it is not measuring confidence.
• The agreement test. Take cases with the correct answer recorded and check whether high confidence really matches being right, and low confidence matches being wrong.

That last test is the only one that truly matters, and the one almost nobody runs, because it requires having cases with the right answer written down. In other words, it requires having built an evaluation suite, which is the piece missing from most projects.

What to do when you discover it measures nothing

Stop showing it. That sounds drastic and it is the correct move: an invented number is worse than no number, because users make decisions with it. In a clinical, legal or financial context, that false precision is a risk in itself.

Then replace it with something honest. What worked for us was showing which criteria were met and which were still missing, so the professional could see the reasoning instead of a score. And when the evidence was not enough, saying so.

The uncomfortable lesson

This was my own system. I built it, I operate it, and I still spent months showing a number I had not verified. That is precisely why I now audit other people's: not because I never make this mistake, but because I know how easily it hides in plain sight, and exactly where to look for it.

If your AI shows a number nobody has validated, tell me about it. And if you want the piece that makes all of this checkable, it is in the AI system audit.