AI confidence scores won't save you from shipping broken products
Original: The Normalization of Inexplicable Failures
Why This Matters
Shows how probability-wrapped AI outputs shift failure accountability from builders to users.
A blog post on ihatethefuture.com argues that AI tools like TypeSafe AI's Jev — which returns typed values with probability estimates — normalize inexplicable failures by giving teams a convenient excuse to ship without proper evals, error budgets, or accountability.
The post opens with a TV show bit: a character blames a stuck door on the door itself, not the obstruction causing it. The author uses this as a frame for how AI products are reshaping expectations around failure. The target is Jev, a model from TypeSafe AI that returns typed responses with attached confidence scores, marketed as fast and cheap. The author's core complaint: to actually verify Jev is working, you need evals and a ground-truth pipeline — which gets you most of the way to fine-tuning your own model anyway. The hard part hasn't gone away; it's just been deferred to the user.
On confidence scores specifically, the author points out two requirements for using them sensibly: understanding how well-calibrated they are, and having a cost model for uncertainty. Jev's marketing leads with benchmark performance, not calibration quality. Their own documentation suggests thresholds of 0.5 and 0.9 for low- and high-risk actions, with a footnote admitting the right values depend on the use case. The author calls this "cargo cult" usage at best.
The broader argument is about accountability. When a web endpoint breaks, there's a defined owner and a traceable contract violation. When an AI call fails, teams can shrug: "AI makes mistakes." That shrug, the author argues, is being actively baked into the product design.