In Search of Confidence

Sooner or later every system we build comes down to one branch:

if confidence > threshold:
    auto_accept(answer)
else:
    queue_for_review(answer)

That branch decides what goes out the door and what costs a human ten minutes, and it rests entirely on the assumption that the number on the left means something. You ask a model how sure it is, it writes a number, and the number looks like a probability because it's a float between zero and one. Nothing about how it got there obliges it to correspond to anything in the world.

Then I found Jev.

Jev is TypeSafe's "System One" model, and it doesn't hand back text. You give it a question and a set of options, and you get a typed decision: a choice, a probability on every option, and a confidence score.

{
  "choice": "cs.CL",
  "probabilities": {
    "cs.CL": 0.78,
    "cs.LG": 0.19,
    "cs.AI": 0.02,
    "cs.IR": 0.01,
    "cs.CV": 0,
    "cs.CR": 0,
    "cs.RO": 0
  },
  "confidence": 0.75
}

I have wanted that object for a long time, so I spent a weekend finding out whether I could believe it.

I walked in with two beliefs I had never tested. The first was that LLMs oversell themselves: ask one how sure it is and it claims more than it has earned. The second was that a probability read from a model's logprobs is more trustworthy than one it writes out as text, and Jev's typed numbers looked like the logprob kind.

Key takeaways

I gave Jev, Claude Sonnet 5 and GPT-5.6 Terra the same 2,000 arXiv papers and the same deliberately ambiguous classification job, then scored their probabilities against what the authors actually chose.

• The frontier models win the headline score. Brier 0.360 for Terra and 0.361 for Sonnet, against Jev's 0.405. They're more accurate too: 76.6% and 75.4% against 73.1%.
• The overconfidence runs the opposite way to what I expected. Jev's average top probability is 0.86, and it's right 73% of the time. Sonnet says 0.67 and is right 75%. Terra, like Sonnet, sits above the line.
• Most of Jev's loss is zeros. On 146 papers it put 0.01 or less on the right answer. Those papers account for its whole gap to Sonnet and three-quarters of its gap to Terra.
• At ranking its own answers, Jev keeps up with the frontier. Sort each model's answers by its confidence and ask how well that order separates right from wrong: Jev 0.785, Sonnet 0.781, Terra 0.765. Jev ties Sonnet and probably has a small edge over Terra.
• It's about 100 times cheaper, answers in about a fifth of a second, and always returned valid typed responses.

Picking a fair fight

Calibration only means something where the truth is uncertain. If the right answer is obvious, a good model says 0.99, is right 99% of the time, and you've learned nothing.

So I needed a classification task that is genuinely ambiguous, and I used arXiv primary categories. When you submit a paper you pick one primary category and may cross-list others. That first pick is a real decision by a real human, and partly arbitrary: a paper on transformers for protein folding could honestly go under machine learning or under AI. A third of the papers I used are cross-listed in two or more of the categories in play. That's exactly the uncertainty I wanted, where the problem isn't that the model doesn't know but that there isn't one right answer.

I picked seven computer-science categories: vision, machine learning, AI, robotics, language, security and information retrieval. From four weeks of submissions (9,983 papers in those categories), I sampled 2,000 submitted between 5 and 17 September. Every model got the same title, abstract and arXiv's own category descriptions, with the options shuffled per paper. Jev received them through its own decisions API rather than a chat prompt with the same content and a different frame.

The scoreboard

The Brier score, squared error on the probability vector, is the right primary metric because it punishes two sins at once: being wrong, and being confidently wrong.

The bottom four are reference points rather than competitors: a TF-IDF classifier trained on 5,503 in-domain papers, an open-weight model read two different ways (more on that later), and an arm that ignores the paper entirely and returns the prior vector every time.

Jev lost. Sort the table by accuracy and you get the same order, so part of what Brier measures here is simply who guesses right more often, and Jev guesses right 3.5 points less often than Terra. Accuracy doesn't explain all of Jev's deficit, though, and the rest turned out to be the most interesting thing in the study. Terra and Sonnet, 0.0016 apart, are tied at the top.

Two jobs for one number

"Confidence" does two jobs, and the scoreboard mixes them. The first is ranking: are your confident answers right more often than your unsure ones? The second is honesty: when you say 0.8, are you right 80% of the time? A model can do either without the other, and people call both of them "calibrated."

To measure ranking alone, sort every paper by the model's stated confidence and ask how often a correct answer outranks a wrong one. That's an AUROC, and overall accuracy doesn't enter into it.

Jev comes first, but only by 0.004 over Sonnet: resample the papers and Sonnet comes out ahead about a third of the time. Against Terra the lead is sturdier. Jev stays ahead in 95% of resamples, just short of the usual bar for calling it real. So on ranking, Jev ties Sonnet and probably has a small edge over Terra.

Honesty is where they split, and it's where my first belief fell apart. In the chart at the top, Jev's curve sits below the diagonal for its whole length. When it says 91%, it's right 69% of the time. It gave 522 papers a top probability of exactly 1.00, and 24 of those were wrong. Sonnet and Terra sit above the line: Sonnet says 63% and is right 74% of the time.

Put the two measures on one chart and the split is plain. Left to right, the three models' ranking scores overlap. Top to bottom, Jev is clearly further from what it claims. The constant baseline in the corner has a lower honesty error than any model on the chart and ranks nothing at all, which is why honesty alone was never going to be the scoreboard.

I assumed LLMs oversell themselves and that a purpose-built model would fix it. On this task it went the other way. Jev oversells, and both frontier models sell themselves short.

Where the loss lives

Jev's probabilities come on a two-decimal grid. Across its 14,000 probability slots (seven per paper) there are 101 distinct values, nothing at all between 0 and 0.01, and 67% of the slots are exactly zero. Sonnet and Terra keep a little probability on categories they doubt. Jev mostly rules them out.

That habit is where the score goes. There are 146 papers where Jev put the true category at 0.01 or below, against 9 for Sonnet and 15 for Terra. Of those 146, 105 were filed under cs.AI, the category every model struggles with. Together they carry Jev's entire gap to Sonnet and 77% of its gap to Terra. On the other 1,854, Jev and Sonnet are level: 0.2958 against 0.2964.

What's interesting is that accuracy doesn't explain this part. On those 146 papers all three models are equally lost: Jev gets none of them right, Terra 0.7%, Sonnet 4.1%. But Jev's Brier there is 1.80 against Sonnet's 1.19, because Sonnet still leaves 0.134 on the right answer on average where Jev leaves 0.005. Identical failure to identify the category, completely different statement about it. Here's one:

Exposing Weaknesses in Emotion Recognition in Conversations

The authors filed it under cs.AI. Jev answered cs.CL with probability 1.00, confidence 1.00, and exactly 0.00 on all six other categories.

I'd probably have said cs.CL too. The 1.00 is what does the damage. The model didn't just miss. It declared the right answer impossible, and it did that on 85 papers. On 52 of those, the category it ruled out was the only one of the seven the authors had tagged at all.

The metric you actually buy

In a real application, nobody consumes a probability vector. You set a threshold.

"Mistakes promised" is what the model's own numbers predict: add up one minus its confidence across everything it accepted. At 0.9, Jev waves through more than half the set, its numbers promise about 27 mistakes in that pile, and there are 133. Terra's promise matches what it delivers almost exactly. Sonnet makes fewer mistakes than it promises.

Or you fix an error budget and let the data choose the threshold. I fixed it at 5%, fit the threshold on four-fifths of the papers, and applied it frozen to the held-out fifth, rotating through all five.

At 5%, Jev hands over the largest share: 28.9% of papers, against 27.9% for Sonnet and 22.7% for Terra. I'd hold this result loosely. The lead over Sonnet is one point, Jev's accepted pile actually ran at 5.7% errors against Sonnet's 5.2%, and 5%, the budget I picked before running anything, turns out to be the only one where Jev comes first:

At 3% Jev isn't beaten, it's lapped, and the reason is the grid. To hit a 3% budget you need to isolate a small, very clean slice of work. Jev's 522 most confident papers all carry the same top probability, 1.00, and that block runs at 4.6% error. Jev can't cut inside the block, so no threshold it can express yields a set cleaner than 4.6%. Ask for 3% and it mostly declines. Sonnet's confidences are fine-grained enough to carve out a small, nearly perfect set that still covers a fifth of the work.

Recalibration

Everything that isn't a zero turns out to be fixable with recalibration.

Temperature scaling divides the log-probabilities by a constant T before renormalizing: above 1 it softens a distribution, below 1 it sharpens one. I fit it on four folds and applied it to the fifth, same as the thresholds.

The direction splits by family. Jev needs T above 1, softening, and so do both Qwen readings at 1.23 and 1.72. Both frontier models need T below 1. They sit so far inside their own claims that the right repair is to make them more confident.

Jev has the most to gain, and its correction is remarkably stable: 1.65, 1.65, 1.64, 1.63 and 1.65 across the five folds. Miscalibration that consistent behaves like a fixed offset, and one number undoes it. That single scalar takes Jev's mean confidence from 0.861 to 0.743, against an accuracy of 0.731, and closes 39% of its Brier deficit against Sonnet and 48% against Terra.

After scaling, Jev is better calibrated than Terra: 0.039 against 0.052. Terra barely moves, because its error was never a simple offset and no single number repairs it.

However, more than half the deficit survives, because the zeros do: on the 146 ruled-out papers, scaling lifts Jev's probability on the right answer only from 0.005 to 0.027. And scaling never changes which label a model picks, so accuracy comes out exactly as it went in.

What about logprobs?

My second belief was that a probability read from a model's logprobs beats one it writes. Written probabilities are prose. Logprobs come straight out of the model.

You can't test that on the frontier models, since none of them expose logprobs on the answer token anymore. So I ran Qwen3.8-27B and asked the same 2,000 questions twice: once "give me a JSON distribution", once "answer with a single letter", then threw the letter away and read the probabilities over the seven letter tokens. Same weights, same backend, same papers. The only difference in the pipeline is the last sentence of the prompt and where I read the number from.

Writing won. Brier 0.516 against 0.578, and when I resample the papers the gap stays between 0.044 and 0.080, nowhere near zero. Accuracy went the same way: 67.1% written against 59.7% from the logprobs.

Part of the reason is that logprobs carry letter preferences along with beliefs. My option shuffle is verifiably uniform: the right answer lands in each slot about 286 times, and a chi-square test finds nothing. The model's top-ranked letters aren't uniform. Position A comes first 367 times and position F only 238 (p ≈ 7 × 10⁻⁸). A preference for A that has nothing to do with the paper leaks into every probability you read off those tokens.

The letter the model actually wrote is noisier still. It lands on G 574 times and F just 86, and it disagrees with the model's own top-ranked letter on 30% of papers. Most of that is sampling: I left the temperature at the provider's default, and drawing from the model's own distribution predicts a 22% disagreement by itself. It doesn't touch the scores above, since I threw the letter away. But if you've built a classifier by asking for a letter at default settings and taking whatever comes back, that 30% is the number to worry about.

That's my second belief gone, at least on the one model where I could test it.

Where Jev is plainly ahead

On cost the gap is large: $0.0988 for all 2,000 papers, against $7.30 for Terra and $10.12 for Sonnet. That's 74 to 102 times cheaper. Jev answered in about a fifth of a second per paper. And all 2,000 of its responses came back complete and correctly typed. Sonnet and Terra needed constrained decoding just to return well-formed JSON, and even then 131 of Sonnet's vectors and 30 of Terra's didn't sum to 1.

Limitations

This experiment is only one task, one model version, and limited to seven of arXiv's 147 categories. Beyond that:

• Jev's training cutoff is an assumption. The papers were submitted between 5 and 17 September, and although unlikely, there could be contamination in Jev's training data.
• Jev saw the task through a different door. It takes structured decisions, not chat, so the categories reached it as criteria in its decisions API. The content was identical. The framing wasn't.
• The labels are soft. If you credit any category the authors tagged and Jev's accuracy goes from 73% to 81%, and its overconfidence shrinks from 0.130 to 0.056.
• The logprob test ran on one quantized model. Both Qwen readings ran on the same fp8 backend, with the sampling temperature left at its default.

Conclusion

I went in wanting a number I could trust and came out with a distinction I'd been missing. "Confidence" has been doing two jobs in my head for years, and I never noticed because on easy tasks the two agree.

Which of these answers should I doubt? and how much should I doubt it? are different questions. Jev keeps up with the frontier on the first and is clearly worse at the second. Sonnet and Terra answer the second well and lean cautious, underselling what they know. Terra, which I've barely mentioned, tied for the top score by being unspectacular at everything. Once you can see the two skills apart, it's more useful to ask which one you need than whether a model is "calibrated."

My second belief fared worse. I assumed better probabilities would come from better plumbing: read the number out of the logprobs, skip the prose, get the truth. The one test I could run, on one open model, says no. The logprobs carried letter preferences that had nothing to do with the papers.

So for routing I need ranking, and Jev is competitive there, though its grid only helps if my error budget sits where the grid can reach. For anything that consumes the number itself I need honesty, and today Terra and Sonnet give me more of it.

That verdict is about jev-1.13 in September 2026. A model at roughly a hundredth of the price came within 3.5 points of the frontier on accuracy and kept up with it at ranking its own answers. Most of what it lost came from a habit of writing 0.00, and both look like things a later version could fix. I think the gap is a version number.

Reproducibility

Everything above is in one repo: the plan I committed before running anything, the dated log of every departure from it, all 12,191 raw API responses from the evaluation run, and the analysis code. Every number and chart in this post regenerates from the committed data without spending anything.

https://github.com/am-shb/llm-confidence