What does a model owe you when it is not sure?

Every accessibility tool I've built generates text, and I'd never thought of that as a design choice.

In a machine shop there's a tool called a go/no-go gauge. It has two ends: one should slide over the part, and the other should not. All it tells you is whether the part fits.

So it can't be persuasive about a part that's out of tolerance. My tools can, and fluently.

The failure I keep writing about on this blog always looks the same. A model reaches for a fact it doesn't have, and it returns a string. It might be an aria-label made from a guess, a contrast ratio that's plausible and wrong, or a spacing token that never existed in the system.

Here's the trouble. A string carries no sign of how sure the thing that wrote it was, so "4.6:1" looks the same whether it was looked up or made up.

So it ships, and the person who finds the difference is the person who can't check. That might be someone with low vision, reading helper text that was never graded. Or a screen reader user, told that a control is something it isn't.

This model gives up writing

On 15 September, TypeSafe AI came out of stealth with Jev (opens in a new tab). They call it a System One model, and the part that interests me isn't something it can do. It's something it refuses to do: Jev gives up string generation entirely.

Instead, you hand it some state and a typed question, and it returns one of the options you supplied, with a probability distribution and a confidence score.

There are three primitives (opens in a new tab). Choice picks from a list of up to 255 options. Score rates the state against a rubric. And Noul answers whether a statement is true, as a number between zero and one.

It can't return something outside the set you gave it, or something malformed. The probabilities are calibrated, which is meant to mean a low number is honest about being low.

None of that makes the model right. What it does is turn the model's doubt into a value my own code can read. That's a different property, and for our work, it's the one worth more.

Accessibility failures are the quiet ones

Our failures are the quiet ones. A wrong contrast value doesn't throw an error, a missing focus style doesn't fail a test, and an invented accessible name doesn't look any different from a researched one.

Nothing in a normal pipeline stops on any of them, and that's why they reach people instead of reviewers.

Now ask the same questions as typed ones. "Which success criterion does this failure fall under?" becomes a Choice across the 86 in WCAG 2.2.

"Does this accessible name describe what the control actually does?" becomes a Noul. And "Does this alt text narrate, or does it merely label?" becomes a Score, against a rubric.

Every one of those comes back with a number attached. So a 0.51 is a flag on a specific line, when today that same doubt arrives as a fluent sentence with nothing on it.

I'd take a check that can say it isn't sure over a check that's usually right, because the first one tells me where to look.

Some things it does not fix, and some I have not measured

Jev can't write the alt text. That's string generation, the thing it gave up, so the generator that invents is still in the pipeline. What's new is that something downstream can grade its output and hand back a number.

The claims are also young, and TypeSafe are more careful about this than the coverage has been. They describe the hallucination rate as zero, and they say plainly that the figure is theoretical, not measured. They note that their own capabilities team built the workflows they tested, and that some bias could exist. And they haven't published benchmarks.

Neither have I run it against anything on this site. I've measured nothing here: this post is a reading of a design decision, not a report on a run. Whether Jev makes any interface more usable, I don't know.

So I'm confident about the approach, not the vendor. Failures that arrive as numbers suit this work better than failures that arrive as prose, whoever ships them.

Here's where I'd point a typed question first.

• Anything you currently trust because it sounds right. Accessible names, alt text and link text leave the least room for doubt, and cost the most when they're wrong.
• Route on the confidence, not only the answer. A low score that goes to a person is a working system.
• Write down what the options are. You can't ask one without first listing the acceptable answers. That shows you which standards were never written down.

The conventions for all this are a week old. What happens to a low confidence score is being decided right now, and so is whether anybody sends it to a human.

Mostly, the people deciding are building real-time control loops for games and drones. Nobody has asked them who gets excluded when a check passes and nobody looks.

So take the check in your pipeline that you trust most because it reads well. Then work out what it would look like as a question with a fixed set of answers and a number on the end.

It costs you an afternoon.