The Audit

Spoke 5 of 6 in the spec-driven development series. Previous: Proving It.

TL;DR. The audit is the part every other guide leaves off, because there’s nothing to sell: it’s a table and ninety minutes. The build can’t check itself, because it verifies against the same model that produced it, so the auditor must be someone, or something, that didn’t build it. Three passes: every criteria against the test that proves it (unmapped criteria is the finding); every built thing against the criteria it traces to (untraced builds are drift, get a verdict); every criteria against what was actually implemented (missed ones expose a broken gate). In the worked example, ninety minutes beat eleven days of green and four approvals, and found a plaintext password leak.

This is the post every other guide to spec-driven development doesn’t write, and I think the reason is simple: there’s nothing to sell here. The audit has no tool, no CLI, no product. It’s a table and a discipline and ninety minutes. So it gets left off the end, and the methodology arrives in your hands with its most important part missing, which is a fitting way for a methodology about missing parts to fail.

Everything up to now has been about doing the work well: forming a good spec, decomposing it, building without drift, proving each test means something. The audit is different. The audit assumes all of that was done conscientiously and by good people, and checks it anyway, because the entire premise of spec-driven development is that conscientious people with good intentions still ship the wrong thing when nothing independent verifies the agreement.

The audit is that independent verification. It’s where you find out whether what got built is what was specified, and it’s where, in the worked example, a plaintext password disclosure got caught in an hour and a half after eleven days of green.

Why the build can’t check itself

Start with the principle, because it’s the thing that makes the audit work and it’s deeply counterintuitive.

The process that built the thing cannot be the process that verifies it. Not because builders are careless, but because verification-by-the-builder is structurally blind. When you build something, you form a model of what it does, and then you check it against that same model, and of course it matches: you’re comparing the thing to your understanding of the thing, and both came from the same place. The bug in test_no_password_in_logs survived because the person who wrote the code wrote the test wrote the belief that the test was sufficient. One mind, one model, internally consistent, wrong.

This is true of humans and it’s true of agents, and it’s especially worth saying about agents because the temptation is enormous. You have an agent that just built the feature. It has all the context. Asking it to verify its own work is right there, one prompt away. And it is close to worthless, because it’s the same model that made the decisions checking whether the decisions were right, which is a model marking its own homework. It will confirm its own understanding with great fluency. That’s not verification. That’s an echo.

So the audit is done by someone, or something, that did not build it. A different person. A different agent, without the build context, given only the spec and the code and told to compare them. The separation is the mechanism. Remove it and the audit becomes theatre, however rigorous it looks.

Part one: what was specified, and what proves it

The audit has three parts, and the first is the traceability matrix, the table that is the whole method’s payoff.

You list every acceptance criteria. Next to each, the test that proves it. Next to that, whether the test is automated, and whether it passes. That’s the table. It sounds trivial. It is the single most revealing artefact in the entire process, because building it forces a question that nothing else asks: for this specific criteria, which test proves it, and does that test actually prove it?

Most of the rows will be fine. Criteria, test, pass, done. Then you hit a row where the test exists and passes and, when you read what it actually asserts, doesn’t test the criteria. That row is test_no_password_in_logs. The matrix is the only place it shows up, because the matrix is the only exercise that puts the criteria and the test’s actual assertions side by side and makes you check that they match. The test suite couldn’t find it; the suite was green. Coverage couldn’t find it; the lines ran. Code review didn’t find it; the code was correct and the test was plausibly named. Only the matrix found it, because only the matrix asked the question the matrix asks.

An unmapped criteria is the finding. Not a paperwork gap, a piece of agreed behaviour that nothing actually verifies. It’s either untested or unbuilt, and the matrix doesn’t care which, because both are the same risk wearing different clothes: behaviour you agreed to and cannot demonstrate you have.

Part two: what got built that nobody asked for

The second part runs the comparison the other direction, and it’s the one that catches drift.

You walk the diff, everything that got built, and for each piece you ask: which criteria does this trace back to? Most trace cleanly. Then you find the ones that trace to nothing. The remember-me checkbox. The password strength meter. The pieces that got built because a capable, literal thing filling in a login form produces the login form it has seen ten thousand times, and nothing compared its output to the agreement until now.

Each untraced piece gets a verdict. Revert it, because it’s on the not-building list and it’s drift. Raise a change request, because actually it’s a good idea and the spec should adopt it deliberately. Or ratify it, occasionally the build’s guess was better than the spec, and the honest move is to update the spec to match, on the record, rather than pretend you planned it.

The verdict distribution tells you something. In the worked example, four untraced pieces: three reverts and one ratify. Three things that shouldn’t have been there and one the spec should have had. That ratio is healthy. If it inverts, if you’re ratifying most of what drifted in, the comfortable reading is “our agent has great instincts” and the honest one is that your specs are too thin to disagree with, and you’re writing them after the fact to match whatever got built, which is documentation, which means you’ve stopped doing this and started narrating it.

And one of those untraced pieces, in the example, was the line that logged the password. It traced to nothing. It violated a criteria and a prohibition simultaneously. It had passed through four code reviews because reviewers read diffs for correctness and this was correct, it was just forbidden, and nothing had been comparing the diff against the list of forbidden things. Part two is that comparison. It’s the gate that the four reviews weren’t.

Part three: what was specified and never came

The third part is the shortest and the easiest to skip, which is why it’s worth doing deliberately. You go back through the criteria and find the ones with no implementation at all.

Some will be legitimately deferred, a criteria marked Should or Could that got consciously pushed to a later sprint, with a decision behind it. Fine. Those are traceable to a choice.

The dangerous row is the one marked “missed.” A criteria that was agreed, tasked, and simply never got built, and nobody noticed until the audit. That’s not a scope decision, it’s a hole, and more importantly it’s evidence that the gate between “task marked done” and “feature actually complete” isn’t working. One missed criteria is a mistake. A pattern of them is a broken process, and part three is where you see the pattern before it sees you.

What ninety minutes bought

Let me total up the worked example, because the numbers are the argument and they’re better than anything I could assert.

The audit took ninety minutes. Going in, the feature had: a full green test suite, ninety-four percent line coverage, four code-review approvals, and eleven days in that state. By every signal a normal team trusts, it was done and it was good.

The audit found a plaintext password disclosure into the log aggregator, the one behaviour the spec said must never happen, that all of those signals had walked past. It found two features built that nobody asked for and that were explicitly on the not-building list. It found one feature that had been built, wasn’t in the spec, and was a good enough idea that the spec should adopt it.

Ninety minutes, against eleven days of false confidence. And the thing that found it wasn’t cleverness or a better tool. It was one person who hadn’t built it, sitting down with the list of criteria, asking for each one: what proves this, and does the proof hold? That’s it. That’s the whole technique. It has no vendor because there’s nothing to charge for, and it’s the only part of this entire methodology that would have caught the thing that actually mattered.

The adversarial habit

Make this a standing practice, not a one-off, and make it genuinely adversarial. The auditor’s job is not to confirm the work is good. It’s to find where the agreement and the reality have come apart, and to approach the work assuming they have, because they usually have somewhere and the only question is where.

If you use an agent for the audit, which works well, precisely because an agent with no build context and a hostile brief is a good drift-finder, give it the spec, give it the code, and tell it to find every place they disagree. Not to assess quality. To find disagreements. A separate agent doing this will find things the building agent was constitutionally incapable of seeing, for the same reason a fresh pair of human eyes does: it never formed the comfortable model that made the bug invisible.

Then take what it finds and give each item a verdict, the same three: revert, change request, ratify. The agent finds the disagreements. A human decides what they mean. That division is right, finding drift is mechanical and tedious and perfect for a machine; deciding whether a given drift should be reverted or adopted is a judgement about the product, and that stays with a person.

The part they left off the end

The reason this post exists is that the audit is the part that makes all the rest worth doing.

You can form a beautiful spec and it’s just a document until something checks reality against it. You can decompose perfectly and build without drift and write tests that pass, and without the audit you have no way to know that what you actually shipped matches what you actually agreed, you have a strong feeling, backed by green ticks, which is exactly the false confidence the whole method was supposed to replace.

The audit is where spec-driven development stops being a nicer way to organise your intentions and becomes a thing that catches the specific, expensive, embarrassing failure that intentions don’t prevent. It’s the difference between a spec you wrote to feel organised and a spec that earns its keep. And it’s left off almost every account of this methodology, because it’s the one part with nothing attached to sell you, which should tell you something about the accounts, and about who’s writing them.

Next, the last one, and the sharpest: The Untold Truth. The tooling is optional, the discipline is free, and the skills you were about to install to automate all this are a genuine security problem that already went off once this year while nobody outside the security press noticed.

The AC Traceability Matrix template is in the action pack. Its worked example is the ninety-minute audit in full, nine criteria, one green tick that was theatre, four untraced builds, and the password in the logs, laid out exactly as it was found.