How Confident Is the AI? Logprobs Explained

When the weather forecast says that there might be a seventy percent chance of rain tomorrow, you grab an umbrella when stepping out of your apartment. You don't need a meteorology degree to be able to use that number to make a decision. You kind of understand instantly that the forecaster is fairly sure but not certain, and you act accordingly.

That little number is doing something subtle and honest. It is not a prediction of rain. It is a confession of uncertainty. The forecaster is telling you not just what they think, but how much to trust what they think.

So here is what I want to convince you of: a language model carries exactly this kind of number, for every single word it produces. And once you can see those numbers, you can build something genuinely useful, which is a system that knows when it is out of its depth and asks for help. That is one of the more important things I built into Ofuma, and it all starts with the humble weather forecast.

What a language model is actually doing

Strip a language model down to its core and here is the loop it runs: it looks at everything written so far and picks the next word. Then it looks again, now including the word it just picked, and chooses the next. One word at a time, over and over. That is genuinely the whole magic trick.

But "picks the next word" is hiding something. The model does not reach blindly into a bag. At each step it weighs thousands of candidate words and gives each one a probability, a sense of "how likely is this the right next word?" Maybe after "The capital of France is" the word "Paris" gets a 90% probability, "Lyon" gets 2%, "a" gets 1%, and the rest split whatever is left. Then it picks, usually the front-runner.

So, buried inside every word the model writes is a number: how sure was I about this one? That is the weather forecast, except there is one for every word. The annoying part is that providers usually hand you the finished sentence and quietly throw the numbers away. But you can ask for them. And when you do, they arrive in a slightly disguised form called a logprob.

What a logprob is, and why the disguise

A logprob (short for log-probability) is just one of those per-word probabilities, run through a logarithm. Before you flinch at the word "logarithm," let me explain why anyone bothered, because the reason is practical and not mathematical vanity.

Probabilities live between 0 and 1. A sentence is many words, and if you want the probability of the whole sentence you end up multiplying a lot of these small numbers together. Multiply enough numbers below 1 and you get something vanishingly tiny. So tiny that a computer, with its finite precision, rounds it to zero and loses the information entirely. It is like trying to weigh a feather, then a feather's shadow, then the shadow of that shadow. Eventually your scale just reads nothing.

Logarithms fix this with a trick you actually met in secondary school: the log of a product equals the sum of the logs. Turn every probability into its logarithm and multiplication becomes addition, and adding a bunch of modest numbers never underflows to zero. Every major model provider (the big ones all do this) reports the log of the probability rather than the probability itself. It is a convention born purely of keeping computers honest.

The practical thing to know is how to read the scale, and I will be honest, it is a little counterintuitive at first:

• A logprob is always zero or negative. Never positive.

• Closer to zero means more confident. A logprob of 0 is total certainty.

• More negative means less confident.

A few landmarks make it concrete:

The mapping back is one operation: raise the mathematical constant e to the power of the logprob, and you recover the original probability. A logprob of -0.22 becomes about 0.80. That's it. The logarithm and its undo are a matched pair. I hope at this point you are not confused 🤔.

Turning many numbers into one

So now, for any answer the model gives, I can get a logprob for every word. A six-word answer is six numbers. That is honestly more than I want to reason about. What I really want is a single confidence score for the whole answer, one number, just like the weather forecast.

The recipe I use in Ofuma is the obvious, honest one. Take each word's logprob, convert it back to a plain probability (the e-to-the-power step from earlier), and then average those probabilities. That average is the confidence score. Roughly: "on average, how sure was the model about each word it chose?"

One small but important detail. I average the real probabilities, not the raw logprobs. Averaging the logs directly would quietly over-punish a single shaky word and drag the whole score down unfairly. Converting back to honest 0-to-1 probabilities first keeps the average fair. The final number gets pinned between 0 and 1 just to keep things tidy.

Let me show it on a sentence the model is sure about. Ask "What is the capital of France?" and it answers, "The capital of France is Paris," with these per-word confidences after converting from logprobs: 0.99, 0.95, 0.99, 0.98, 0.99, and 0.91 for "Paris." Average them and you get about 0.97. Ninety-seven percent confident. This is a model standing on solid ground, and it shows.

Now a harder case. Ask something genuinely ambiguous and watch the model start hedging: "The answer might be approximately 42." Word by word, its confidence sags in the telltale places. It was only about 30% sure of "might" and a mere 8% sure of "42." Average the whole thing and you land near 0.59. Fifty-nine percent. The model is, quite literally, not sure what it is saying, and the number captures it. And notice where the confidence collapses: on the very words carrying the real meaning, the hedge and the final figure. The score is reading the model's hesitation exactly where it matters.

A second opinion: folding in guardrails

Word-level confidence is a strong signal, but it only measures one thing, which is how sure the model was about its phrasing. It says nothing about whether the answer was actually good. A model can be supremely confident and confidently wrong. We have all met that person 😅.

So Ofuma blends in a second signal when it is available: the results of the guardrails attached to that prompt, which are the policy checks that fire on the output and come back pass or fail. If three of four guardrails pass, that is a 0.75 from the quality side of the house.

When both signals exist, I weight them 70% toward the logprob confidence and 30% toward the guardrails. The model's own probabilities get the larger share because they come straight from its internal certainty, which is the most direct read we have. The guardrails are a binary quality check, a valuable but secondary witness. If only one signal is available, it carries the full vote. And if neither exists, the system assumes confidence rather than crying wolf.

A quick blend so it is not abstract. Suppose the logprob side works out to 0.94 and three of four guardrails pass for 0.75. The final score is 0.94 x 0.7 + 0.75 x 0.3, which lands around 0.88. One number, drawn from two independent sources of doubt.

The point of all this: knowing when to ask for help

Here is where the number finally earns its keep.

Every prompt in Ofuma can carry a confidence threshold, a line in the sand, say 0.70. After the model answers, the system computes that blended confidence score and holds it up against the threshold. Above the line, the answer goes out the door. Below it, the system does not simply shrug and ship a shaky answer. It does something about it, and you get to choose what:

• Do nothing: accept the low-confidence answer, but at least you logged that it was shaky.

• Fall back to a safe canned response: swap in a pre-written reply you know is harmless.

• Retry with a different model: give a stronger model a swing at it.

• Escalate to a human: hand the case to a person, along with the input, the output, and the confidence score, so they can approve it, fix it, or reject it.

That last option is the one that matters most to me, because it is the whole reason for the exercise. The dream is not an AI that is never wrong, because that does not exist. The dream is an AI that knows when it might be wrong and has the good sense to raise its hand. The confidence score is what lets it raise its hand. When the number dips under the threshold, a human gets pulled in precisely on the cases that need a human, and only those, so nobody wastes their day reviewing answers the model already nailed. (I would love to talk about how the human review queue itself works in another article perhaps 🙂.)

Back to the umbrella

We started with a forecaster admitting seventy percent. The reason that number is useful is not that it is always right. It is that it is honest about its own uncertainty, which lets you decide what to do.

A logprob is that same confession, whispered by a language model for every word it writes. Most systems never listen for it. The ones that do can finally build the thing everybody actually wants from AI: not a machine that pretends to be certain, but one that tells you, plainly, when to grab an umbrella, or when to go find a human.

Where no counsel is, the people fall: but in the multitude of counsellors there is safety. - Proverbs 11:14