Who verifies what the agents built?

Every one of us has felt this in our own week: the pull request that arrives complete, plausible, and larger than there is time to read properly.

Generation got faster. Review did not. The distance between those two things is not a productivity statistic. It is a gap, and things fall into gaps.

What actually falls into that gap?

Accessibility defects, more than most.

A wrong contrast pair does not break a test. A focus style that was never written does not throw. An aria-label invented from a guess announces itself with total confidence, and the reviewer skimming a two-thousand-line diff has no reason to stop on it. It looks exactly like the code around it.

So the person who finds it is not the reviewer. It is someone with low vision who cannot read the result, or a keyboard user who loses their place, or a screen reader user told something about a control that is not true.

That is the 2021 argument again, standing on new ground. Exclusion is the default outcome of unexamined defaults. We have simply acquired a machine that produces defaults at a rate our review practices were never designed to absorb.

The obvious answer is to slow generation down. I do not believe that works, and I do not believe anyone will do it. So the proposal I would make instead is this: do not slow generation down; make verification cheap, visible, and explicitly somebody's job.

Can a thing check its own work?

Not usefully. An agent grading its own homework is not a check, it is a second opinion from the same source.

That is the design principle I built atomic-cockpit (opens in a new tab) around: a control surface for running many agents across several projects at once, built on Atomic, the open-source agent runner.

You say what you want built. An agent writes a plan. Nothing happens until a person approves it. Then it hires the agents the job actually needs. A small job gets a small team. They build, they talk to each other, and a different agent, one that did not write the code, verifies the result against the checks attached to the plan.

Around that sit the parts that let one person follow a team of agents: what each agent is saying to the others, a board of work in flight, a roster of who is hired and on what.

The separation is the whole point. A checker working from a plan it did not write is a real check. Anything else is an expensive way of agreeing with ourselves.

Which checks do I actually trust?

Three properties, and most of what we call review has only one.

Independence. The checker did not produce the thing. This is the property agents make easy and humans make expensive, which is a genuine reversal: a second reviewer used to cost a salary and now costs pennies.

A prior definition of success. Checks written before the work started get run. Checks remembered at the end get skipped when the diff is large, which is exactly when they matter. Attaching them to the plan rather than the review is the whole difference.

A person can follow it. A verification nobody can inspect is a claim, not a check. This is why the roster and the message board exist, not as features, but so one person can see what a team of agents actually did.

Accessibility checks fail the second property more than any other. They are almost never in the definition of done, so they are almost never in the review.

What did one real run cost?

The case study (opens in a new tab) documents a run rather than a demo. Nine checks, all passing, verified by an agent that did not write the code, then reviewed by a person. Fourteen minutes from approval to proof. One dollar ninety-eight.

Those are measurements, and they are the only claims I will make here. Whether this setup produces more accessible software than the alternative is not something that run establishes, and I have not measured it.

The cost line is the interesting one. Two dollars is small enough that "try it and see" has changed, and large enough that running fifty carelessly is a real number. Both are true and we tend to hold only one at a time.

That run also surfaced a defect in one of the cockpit's own tabs, which stayed in the write-up. A tool that only ever shows success is not being tested. It is being marketed. Every demo any of us has watched was a run somebody chose, and the useful information is in the runs they did not. The screenshots were taken every two seconds while it happened rather than staged afterwards, and there is a commit in the history titled "describe the screenshots accurately instead of retouching them."

What can we put in the gap on Monday?

None of this requires the same tooling. The moves are portable.

• Separate the verifier from the author. A different agent, a different person, a checklist written before the work started, anything that is not the thing that produced the output.
• Attach checks to the plan, not the review. Checks agreed up front get run.
• Put accessibility on that list by name. Contrast, focus order, accessible names, behavior at 375 pixels. If they are not on the list they are not in the review, because they do not fail loudly.
• Keep the approval gate, especially when it annoys you. It is friction. It is also the only moment where someone with context sees the plan before tokens start moving.
• Publish the run that went wrong. Cheap if decided in advance, expensive if decided under pressure.

The conventions for how generated work gets verified are being set now, while they are soft, mostly by people who have not been asked who gets excluded when nobody checks.

Let's be the ones who put the check in the plan.