Who verifies what the agents built?

A pull request arrives complete and plausible, and it's bigger than there is time to read properly. Every one of us has felt that in our own week.

Generation got faster, but review didn't. I don't read the distance between them as a productivity statistic. It's a gap, and defects fall into it.

Accessibility defects fall into the gap

And they fall into it more than most defects do.

A wrong contrast pair doesn't break a test. A focus style that was never written doesn't throw an error. And an aria-label invented from a guess announces itself with total confidence.

A reviewer skimming a two-thousand-line diff has no reason to stop on it, because it looks exactly like the code around it. So the reviewer isn't the one who finds it.

It's found by someone with low vision who can't read the result. Or a keyboard user who loses their place. Or a screen reader user told something untrue about a control.

That's the 2021 argument again, on new ground: exclusion is the default outcome of unexamined defaults. Only now we have a machine that produces defaults faster than our review practices were ever built to absorb.

The obvious answer is to slow generation down. I don't believe that works, and I don't believe anyone will do it. So I'd leave generation at the speed it is, and make checking cheap, visible and explicitly somebody's job.

That somebody can't usefully be the agent that did the work. An agent grading its own homework is a second opinion from the same source, and I don't count that as a check.

That's the principle I built atomic-cockpit (opens in a new tab) around. It's a control surface for running many agents across several projects at once, built on Atomic, the open-source agent runner.

Here's how it goes. You say what you want built, and an agent writes a plan. Nothing happens until a person approves it. Then it hires the agents the job actually needs, so a small job gets a small team.

They build, and they talk to each other. Then a different agent, one that did not write the code, checks the result against the checks attached to the plan.

A checker working from a plan it didn't write is a real check. Without that separation, we're paying to agree with ourselves.

The case study (opens in a new tab) documents a run, not a demo. It had nine checks, all passing, verified by an agent that did not write the code and then reviewed by a person. It took fourteen minutes from approval to proof, and it cost one dollar ninety-eight.

Those are measurements, and they're the only claims I'll make here. Does this setup produce more accessible software than the alternative? I don't know. That run doesn't show it, and I haven't measured it.

The cost is the part I find interesting. Two dollars is small enough that "try it and see" has changed. But it's also big enough that running fifty carelessly is a real number, and we tend to hold only one of those thoughts at a time.

That run also turned up a defect in one of the cockpit's own tabs. Of course it did. And the defect stayed in the write-up, because a tool that only ever shows success is being marketed, and nobody is testing it.

Think about every demo any of us has watched. Each was a run somebody chose, and the useful information is in the runs they didn't choose.

A check worth trusting has three properties

None of this needs the same tools. When I decide whether to trust a check, I look for three things, and most of what we call review has only one.

The first is independence: the checker didn't make the thing. Agents make this easy where people make it expensive. A second reviewer used to cost a salary, and now it costs pennies.

The second is a definition of success, written first. Checks written before the work started get run. Checks remembered at the end get skipped when the diff is large, which is exactly when they matter most.

The third is that a person can follow it. If nobody can inspect a check, it's only a claim. That's why the cockpit's roster and message board exist: so one person can see what a team of agents actually did.

Accessibility checks fail the second test more than any other. They're almost never in the definition of done, so they're almost never in the review.

• Separate the verifier from the author. Use a different agent, a different person or a checklist written before the work started. Anything will do, as long as it isn't the thing that produced the output.
• Attach checks to the plan, not the review, and put accessibility on that list by name. Contrast, focus order, accessible names, behavior at 375 pixels. They don't fail loudly. If they aren't on the list, they aren't in the review.
• Keep the approval gate, especially when it annoys you. It's friction. It's also the only moment when someone with context sees the plan before tokens start moving.
• Publish the run that went wrong. That's cheap if we decide it in advance, and expensive if we decide it under pressure.

The conventions for how generated work gets checked are being set now, while they're soft. And most of the people setting them haven't been asked who gets excluded when nobody checks.

One last thing, on that last item. The screenshots in the case study were taken every two seconds while the run happened, not staged afterwards. There's a commit in the history titled "describe the screenshots accurately instead of retouching them."