Proving It

Spoke 4 of 6 in the spec-driven development series. Previous: Building Without Drift.

TL;DR. Coverage tells you which lines ran, not whether anything checked them, and you can have ninety-four percent coverage sitting next to a plaintext password leak. Don’t test the code, test the acceptance criteria: every one gets a test that asserts the behaviour it describes, awkward corners included. Assert behaviour, not implementation, so a refactor doesn’t break it and a real bug can’t slip past. Watch every test fail on purpose before you trust it. And write the negative tests nobody writes, for the things that must never happen, on the exception paths especially. A green suite is a pile of claims; verification is checking which are true.

There’s a test in the worked examples called test_no_password_in_logs. It passed. It passed for eleven days. It was green in the pull request, green in CI, green on the dashboard everyone glanced at.

And the whole time, sitting in the exception handler, was a line that wrote the full login request, password included, in plaintext, straight into the log aggregator. The one thing the criteria said must never happen. Happening. Under a green tick named after the exact thing it wasn’t checking.

That test is what this post is about, because it’s the most dangerous artefact in software: a test that passes and proves nothing. It’s worse than no test at all, and understanding exactly why is the difference between verification that means something and verification that’s theatre with a coverage number.

Coverage is not verification

Let’s kill the most common confusion first, because it’s load-bearing for a lot of false confidence.

Coverage tells you which lines of code ran while the tests executed. That’s all it tells you. It does not tell you whether those lines were checked, whether anything asserted they did the right thing. A test can execute a line, observe nothing about it, and coverage counts it as covered. You can have ninety-four percent coverage and near-zero verification, and that combination isn’t a freak occurrence, it’s a genre. The login example had ninety-four percent coverage. It also had a plaintext password disclosure. Those facts sat together comfortably, because coverage measures execution and the bug was about assertion.

So when someone tells you the code is “well tested” and reaches for a coverage number, they’ve answered a different question from the one that matters. The question that matters is not “did the tests touch this code.” It’s “does a test prove this criteria holds, and would it fail if the criteria were violated.” Coverage cannot see that. Only reading the assertions can.

Test the criteria, not the code

Here’s the shift that makes verification real. You are not testing the code. You are testing the acceptance criteria, and the code is merely where they happen to live.

This sounds like a distinction without a difference until you watch it play out. A test written to test the code asks “does this function do what it does,” which is close to meaningless, because the function does what it does by definition. A test written to test the criteria asks “does the system exhibit the behaviour we agreed on,” and that’s a question with a real answer that can be no.

Every acceptance criteria from your feature spec should have a test that asserts it. Not gestures at it. Asserts it. The criteria says five failed attempts lock the account for fifteen minutes; the test creates five failed attempts, tries a sixth with the correct password, asserts it fails, and asserts the lock is still there. It tests the behaviour the criteria describes, including the awkward corner, the correct password during the lock, that the naive test skips because it feels redundant.

When your tests are written against criteria, verification becomes the mechanical thing it should be. Here’s a criteria, here’s the test that asserts it, the test passes, the criteria holds. The chain is legible. Anyone can follow it. And crucially, anyone can spot where it breaks, which is what a traceability matrix does, and that’s the next post.

Behaviour, not implementation

There’s a trap inside “test the criteria,” and it’s worth naming because good engineers fall into it constantly.

Test what the system does, not how it does it. A criteria describes observable behaviour: given this, when that, then this result. The test should assert the result, not the mechanism. The moment your test knows that the password is hashed with a particular library in a particular function, it’s testing the implementation, and it will break the instant someone refactors, break without anything actually being wrong, which trains everyone to ignore it, which is how you end up with a suite people mute.

A test that asserts behaviour survives refactoring, because the behaviour is the contract and the implementation is just today’s way of honouring it. A test that asserts implementation is a tripwire in your own house. It goes off when you move furniture, teaches you to stop listening, and then doesn’t go off when the burglar arrives because you’ve disabled it out of habit.

The discipline: write the test so it would pass against any correct implementation and fail against any incorrect one. If a legitimate refactor breaks your test, your test was wrong. If a behavioural bug slips past it, your test was wrong. Both failures point the same direction, the test was coupled to how instead of what.

The test that passes and lies

Now back to test_no_password_in_logs, because it deserves a full autopsy. It’s the whole point.

The criteria was clear: no password reaches the logs on any login attempt, including one that throws an exception. Three paths, named explicitly. The test covered two of them, the successful login and the wrong-password case. Both are handled paths; neither throws. The third path, the exception path, was the only one where a serialised request body could plausibly hit a logger, and it was the one the test skipped, because forcing an unhandled exception in a login test felt contrived and nobody wanted to write it.

So the test asserted that passwords didn’t leak on the two paths where they were never going to, and stayed silent about the one where they did. It passed. Honestly, in a sense, every assertion in it was true. It just wasn’t asserting the thing that mattered.

Then four people reviewed the code, and each of them saw a test named test_no_password_in_logs, and read the name, and believed it. The name was doing work the assertions weren’t. That’s the mechanism. A test’s name is a claim, and a claim is not evidence, and the gap between them is where this class of bug lives, invisible precisely because everything is green and everything is named reassuringly.

The lesson isn’t “write more tests.” It’s that a test is only worth the assertion inside it, and the only way to know what a test proves is to read what it asserts, not what it’s called. A green suite is a collection of claims. Some of them are true. Verification is the work of checking which.

Making tests fail on purpose

There’s one practice that catches more of this than any other, and it’s almost never done: watch every test fail before you trust it passing.

Write the test. Run it. It’s green. Now break the thing it’s supposed to be checking, comment out the lock logic, remove the guard, invert the condition, and run it again. If it’s still green, the test is theatre, and you’ve just found out for the price of thirty seconds instead of eleven days. Put the code back, watch it go green again, and now you know the green means something.

This is the single highest-leverage habit in verification, and it’s rare because it feels backwards, you’re trying to make your own tests fail, which cuts against every instinct that wants the suite green so you can go home. But a test you’ve never seen fail is a test you don’t actually know works. The test_no_password_in_logs test would have survived this check, because it did pass when things were fine and would have failed if you’d broken the two paths it covered. Which tells you the technique isn’t sufficient on its own, you also have to check the test covers the right paths. But most theatre tests don’t even survive being broken, and this catches those instantly.

The negative cases nobody writes

Your “what must never happen” list needs tests too, and they’re the hardest to write because they’re backwards. A normal test asserts something happens. A negative test asserts something doesn’t, no password in the logs, no cross-tenant access, no different error for missing versus wrong.

Nobody writes these naturally, because the happy path is what you’re building and the happy path is where your attention is. The prohibition tests require you to actively imagine the system misbehaving and assert that it doesn’t, and to do it for the specific misbehaviours you called out as unacceptable, on the specific paths where they’d actually occur. Including the exception path. Especially the exception path, because that’s where the guards you put on the happy path quietly don’t apply.

If your spec has a “must never happen” section, and it should, then every line in it needs a test that would go red if the forbidden thing occurred. Those are the tests that stop the newspaper headline. They’re also the tests most likely to be missing, because they protect against things that haven’t happened yet and feel paranoid to write, right up until the ninety minutes where one of them would have saved you.

What proving it actually buys you

At the end of real verification you don’t have “the code is tested.” You have something far more specific and far more defensible: every criteria has a test that asserts it, every test asserts behaviour rather than implementation, every test has been seen to fail when the behaviour breaks, and every prohibition has a negative test on the paths where it could actually be violated.

That’s a claim you can stand behind, because each piece of it is checkable. “Tested” is a feeling. “Every criteria has an asserting test that fails when the behaviour breaks” is a fact, and it’s a fact someone who wasn’t in the room can confirm, which is the entire bar this whole method is built around.

But notice what verification still can’t do on its own. It can tell you each test you wrote passes and means something. It cannot tell you whether you wrote a test for every criteria, whether something got built that no criteria covers, or whether a criteria quietly never got implemented at all. For that you need to step outside the tests and compare the whole thing, specified against built against proven, from a position the build itself can’t occupy.

That’s the audit, and it’s the last post, and it’s the one that found the plaintext password in ninety minutes when everything else had been green for eleven days. The Audit.

The Definition of Done template in the action pack is where verification stops being vibes: every line checkable by someone who wasn’t in the room, and a banned-words list that starts with “tested,” because “tested” was true the whole time the passwords were leaking.