In Search of Confidence

Sooner or later every system we build comes down to one branch:

if confidence > threshold:
    auto_accept(answer)
else:
    queue_for_review(answer)

That branch decides what goes out the door and what costs a human ten minutes, and it rests entirely on the assumption that the number on the left means something. You ask a model how sure it is, it writes a number, and the number looks like a probability because it's a float between zero and one. Nothing about how it got there obliges it to correspond to anything in the world.

Then I found Jev.

Jev is TypeSafe's "System One" model, and it doesn't hand back text. You give it a question and a set of options, and you get a typed decision: a choice, a probability on every option, and a confidence score.

{
  "choice": "cs.CL",
  "probabilities": {
    "cs.CL": 0.78,
    "cs.LG": 0.19,
    "cs.AI": 0.02,
    "cs.IR": 0.01,
    "cs.CV": 0,
    "cs.CR": 0,
    "cs.RO": 0
  },
  "confidence": 0.75
}

Those numbers aren't something the model wrote. They're something it emitted. I have wanted that object for a long time, so I spent a weekend finding out whether I could believe it.

Key Takeaways

I gave Jev, Claude Sonnet 5 and GPT-5.6 Terra the same 2,000 papers and the same deliberately ambiguous classification job, then scored their probabilities against what happened. Alongside them ran a supervised classifier, an open-weight model, and a baseline.

• Terra and Sonnet still win the headline score. Brier 0.360 and 0.361 against Jev's 0.405, and they're more accurate too, 76.6% and 75.4% against 73.1%.
• Jev is the best at ranking the options. Rank every paper by stated probabilities and count how often a the model was right: Jev 0.785, Sonnet 0.781, Terra 0.765.
• The results turned the opposite way of what I expected. Jev says 86 and is right 73. Sonnet says 67 and is right 75. The vague thing I believed walking in, that models oversell themselves, holds for Jev and reverses for both frontier models.
• Jev does not say "unlikely, but possible." Almost all of Jev's gap comes from 7% of papers where it drove the right answer to zero.
• Where Jev is plainly ahead: 74x to 102x cheaper, about a fifth of a second per answer, and 2,000 correctly typed responses out of 2,000 calls.

If you want to know the whole story, read on.

Picking a fair fight

Calibration only means something where the truth is uncertain. If the right answer is obvious, a good model says 0.99, is right 99% of the time, and you've learned nothing.

So I had to come up with a genuinely ambiguous classification task. I used arXiv primary categories. When you submit a paper you pick one primary category and may cross-list others. That first pick is a real decision by a real human, recorded permanently, and partly arbitrary: a paper on transformers for protein folding could honestly go under machine learning or under AI. Exactly the uncertainty I wanted, which is not "the model doesn't know" but "there isn't one right answer."

I pulled 9,983 papers from a 30-day window, took the seven biggest CS categories and sampled 2,000. Every model got the same title, abstract and arXiv's own category descriptions, with options shuffled per paper.

The scoreboard (and what I didn't expect)

The Brier score, squared error on the probability vector, is the right primary metric because it punishes two sins at once: being wrong, and being confidently wrong.

The bottom four are reference points rather than competitors: a TF-IDF classifier trained on 5,503 in-domain papers, an open-weight model read two different ways (more on that later), and an arm that ignores the paper entirely and returns the prior vector every time.

Jev lost. Except that "lost" is carrying more weight than it deserves. Rank these six arms by Brier, then rank them by accuracy. The scoreboard is mostly measuring which model guesses right — and Jev guesses right 3.5 points less often than Terra. That's a real gap, and it isn't a statement about its probabilities.

The deeper problem is that "confidence" is doing two jobs at once and the scoreboard only measures one of them. Discrimination is knowing which papers are hard: do your high-confidence answers differ in outcome from your low-confidence ones? Honesty is the level: when you say 0.8, does it happen 80% of the time? A model can have either without the other, and "calibrated" gets used for both.

To ask which model has the best estimate of its own confidence you need a question accuracy can't answer: sort every paper by the model's stated confidence, then ask how often a correct answer outranks a wrong one. That's an AUROC, and the overall hit rate doesn't enter into it.

Jev is furthest right and sits high: it knows which papers are hard, and it's worse at telling you by how much. That's why the scoreboard and the coverage bars in the next section disagree without contradicting. Coverage only inspects the accepted set, so it rewards ordering and is blind to whether the numbers mean anything. Brier integrates over every paper, including the catastrophes.

The shape of that loss is in the first chart. Jev's curve sits below the line for its whole length: when it says 91, it's right 69% of the time, and 443 papers came back at a stated confidence of exactly 1.000, 17 of them wrong. Sonnet sits above the line, saying 63 and getting 74%.

I walked in assuming LLMs oversell themselves and that a purpose-built one would fix it. Here the frontier models are the ones selling themselves short.

The metric you actually buy

In a real application, nobody consumes a probability vector. Instead, you set a threshold.

Or sometimes you want to fix an error budget and let the data choose the threshold. I fixed it at 5%, fit on four-fifths of the papers, applied it frozen to the held-out fifth.

Jev hands over the largest share, and this is the result I'd hold most loosely in the study. And the win doesn't survive moving the budget. 5% is the budget I'd picked before running anything, and it turns out to be the only one where Jev comes first:

At 3% it isn't beaten, it's lapped, and that comes straight back to the grid. To hit a 3% budget you need to isolate a small, very clean slice of work. Jev's 522 most confident papers all carry one identical confidence value, and that block runs at 4.6% error. It cannot cut inside the block, so no threshold it can express produces a set cleaner than 4.6%. Ask for 3% and it mostly declines. Sonnet's confidences are fine-grained enough to carve out a tiny near-perfect set and still keep a fifth of the workload.

On cost the gap isn't subtle: $0.0988 for all 2,000 papers against $7.30 for Terra and $10.12 for Sonnet, with a correctly typed response every single time. The other arms needed constrained decoding to get there and still returned 131 responses whose probabilities didn't sum to 1.

Where the loss lives

There are 146 papers where Jev put the true category at 0.01 or below, against 9 for Sonnet and 15 for Terra. Sonnet and Terra have a vocabulary for "probably not, but check back." Where as Jev has zeroed 67% of the 14,000 slots. On the other 1,854 papers Jev is level with Sonnet, 0.2958 against 0.2955. Those 146 carry 99% of its gap against Sonnet and 77% against Terra.

This part isn't the accuracy effect in disguise. On those 146 papers all three models are equally lost: Jev gets 0% of them right, Terra 0.7%, Sonnet 4.1%. Nobody can classify them. But Jev's Brier there is 1.798 against Sonnet's 1.184, because Sonnet still leaves 0.134 on the right answer where Jev leaves 0.005. Identical failure to identify the category, completely different statement about it. Here's one:

Exposing Weaknesses in Emotion Recognition in Conversations

The authors filed it under cs.AI. Jev answered cs.CL with probability 1.00, confidence 1.00, and exactly 0.00 on all six other categories.

I'd probably have said cs.CL too. The 1.00 is what does the damage. The model didn't just miss, it declared the right answer impossible, on 146 papers, and on 52 of them the category it zeroed was the only one the authors had tagged.

Recalibration

Everything that isn't a zero turns out to be improvable with a recalibration.

Temperature scaling divides the log-probabilities by a constant T before renormalizing: above 1 it softens a distribution, below 1 it sharpens one. I fit it on four folds and applied it to the fifth, same as the thresholds.

The direction splits by family. Jev needs T > 1, softening, and so do both Qwen readings at 1.23 and 1.72. Both frontier models need T < 1. They sit so far inside their own claims that the correct repair is to make them more confident.

Jev has the most to gain, and its offset is remarkably stable. Its temperature came out 1.65, 1.65, 1.64, 1.63, 1.65 across the five folds. Miscalibration that consistent is a fixed offset rather than confusion. One scalar takes Jev's mean confidence from 0.861 to 0.743 against an accuracy of 0.731, and closes 39% of its Brier deficit against Sonnet and 49% against Terra.

And after scaling, Jev is better calibrated than the model that won the study, 0.039 against Terra's 0.052. Terra barely moves, because its error was never a simple offset and no single number repairs it.

Two things keep this from being a rescue. Half the deficit survives, because the zeros do. And scaling never changes which label a model picks, since the argmax is untouched, so accuracy, coverage and ordering come out exactly as they went in. It fixes the numbers and leaves everything else alone.

What about logprobs?

My excitement rested on an intuition I'd never tested: that a probability a model emits beats one it writes. Written probabilities are prose; emitted ones come from the machinery.

You can't test that on the frontier models, since none expose logprobs on the answer token anymore. So I ran Qwen3.8-27B and asked the same 2,000 questions twice: once "give me a JSON distribution", once "answer with a single letter", then threw the letter away and read the logprobs over the seven letter tokens. Same weights, same backend, same papers. The only difference in the pipeline is the last sentence of the prompt and where I read the number from.

Writing won. Brier 0.516 against 0.578, bootstrap interval [0.044, 0.080], comfortably excluding zero. Accuracy went the same way: 67.1% written against 59.7% from the logits.

The logprobs encode letter preferences alongside beliefs. My option shuffle is verifiably uniform: the right answer lands in each slot about 286 times, chi-square p = 0.56. The model's picks aren't. Position A wins 367 times, position F 238 (p ≈ 7 × 10⁻⁸). Worse, the token it actually spoke lands on G 574 times and F just 86. The spoken letter disagrees with its own top-ranked letter on 30% of papers. If you've ever built a classifier by asking for a letter and taking what comes back, sit with that number.

Limitations

One task, one model version, seven of arXiv's 147 categories, a single 48-hour window. This study doesn't come without limitations:

• The labels are soft. Credit any category the authors tagged and Jev's accuracy goes from 73% to 81%, its confidence gap from +0.109 to +0.035. Some of what I'm calling overconfidence is the task being underdetermined.
• The hard class is hard for everyone. Every model under-predicts cs.AI. Jev is most extreme but not alone, so I'm partly measuring the taxonomy. Where it is distinctive: on papers filed cs.AI, its top pick is cs.CL (117) or cs.LG (113) more often than cs.AI (93). Both frontier models put cs.AI first.

Conclusion

I went in wanting a number I could trust and came out with something more useful. "Confidence" has been doing two jobs in my head for years, and I never noticed because on easy tasks the two agree.

Which of these should I doubt? and how much should I doubt it? are different questions. Jev is excellent at the first and bad at the second. Sonnet and Terra are the reverse, with well-behaved numbers, weaker ordering, and a habit of hedging. Terra, which I've barely mentioned, won the study by being unspectacular at everything. Once you can see the two skills apart, "is this model calibrated?" stops being the right question and which of the two do I need? takes its place.

The second thing I'd keep is less comfortable. I assumed better probabilities would come from better plumbing: read the number out of the logits, skip the prose, get the truth. The one clean test says no. The plumbing carries letter preferences, sampler noise, and whatever the model was thinking about thirty tokens ago. Where a number comes out of a model matters far less than whether the model was ever taught to make it mean something.

So for routing I need ordering, and Jev is competitive there, though only far enough to matter if my error budget sits where its grid can reach. For anything that consumes the number itself I need reliability, and today Terra and Sonnet give me more of it. Against that, Jev costs a hundredth as much, answers in a fifth of a second, and returned 2,000 valid typed responses out of 2,000 calls while the others needed constrained decoding and still sent back 131 vectors that didn't sum to 1.

That's a verdict about September 2026 rather than about the idea. A model at a hundredth of the price came within three and a half points of the frontier on accuracy, matched both frontier models at knowing which of its own answers to doubt, and did it on its first release. I don't think that gap is structural. I think it's a version number.

Reproducibility

Everything above is in one repo: the protocol, all 12,191 raw API responses, the analysis code, and every decision I made along the way.