From coverage to confidence

When I go into organizations, I spend a lot of time talking with teams about quality. I'll ask a pretty simple question: how do you know the software you're putting into production is good? How do you know it's safe?

Usually people start talking about their regression tests, so I'll ask about test coverage. What's good coverage? Most of the time I hear something around 80 percent. In a lot of organizations, 80 is the number management asked for.

The question after the question

So let's say you have 83 percent coverage. That sounds pretty good. Then I ask the question I think matters much more.

How do you know you have the right tests?

That's usually where the conversation gets interesting. The answer is often, "Well, we add more tests." But more tests isn't necessarily the answer. What I want to know is whether the tests are actually protecting the behavior of the software.

Flip a conditional and watch what happens

Before I even introduce a mutation testing framework, I'll often do it by hand with the team. I'll ask a developer to make a tiny change in the production code — flip a conditional, change a magic number, change a return value.

Then we run the tests. Everything stays green.

I'll ask the team, "Is that a good thing?" Usually somebody says yes. All the tests passed.

And I'll say, "But we just changed the behavior of the production code, and none of our tests noticed."

That's usually the moment it clicks. We just learned something about the quality of our tests that coverage alone never told us.

From there we can look at frameworks that do this systematically: small, controlled changes throughout the code, tests run against each one. If the tests fail, the mutant was killed. If the tests stay green, we have a living mutant.

Coverage is not confidence

That's a very different conversation than test coverage. Coverage tells me the code was exercised. Mutation testing tells me whether the tests would actually notice if the behavior changed.

Picture a team sitting at 83 percent coverage. They've exceeded management's 80 percent target and everybody feels pretty good about it. Then they run mutation testing and the score comes back at 52 percent.

Same code. Different truth.

Now we have something worth talking about.

And the answer isn't necessarily more tests. It's better tests — tests that actually detect changes in behavior. That's one of the things I really like about mutation testing. Coverage may barely move while the mutation score climbs a long way, because we're strengthening the tests around code that was already covered.

In a scenario like that, coverage might crawl from 79 to 82 while the mutation score travels from the low 50s into the high 90s. The numbers below are drawn to show that shape; they are illustrative, not measured on a ClickChain project. That is a very different improvement in quality than three points of coverage suggests.

Why is this only on a dashboard?

And this is where I start thinking about governance. If we can produce this evidence automatically, why would we only put it on a dashboard?

Why isn't it a gate?

Especially now that AI can write code with incredible speed. The question isn't how to make the agent write code faster. It's how to keep that speed while building quality into the work as it happens. Test-first development, mutation testing, and evidence gates add some work to each cycle, but with agentic coding the cycle is still wicked fast. Now we're getting speed with evidence, instead of speed followed by cleanup, rework, and defects.

An agent can write code, write tests, open a pull request, and everything can be green. Great. But I'm still going to ask: are they the right tests?

Green tests alone don't give me much confidence. I want the tests challenged. That's what mutation testing does. Run the mutations. Are there living mutants? If there are, the gate holds. Address the weakness, generate the evidence again, then move forward.

That, to me, is much more interesting than another approval process.

Governance inside the cycle

And I don't think we should wait until the pull request. I want the governance inside the development cycle, while the software is being created — not added afterward as another quality gate.

If we're working test first, the discipline starts there: one failing test, make it pass, refactor, challenge the tests, produce the evidence, then move to the next behavior. That's what I mean by shifting governance left. We're not trying to inspect quality into the software at the end. We're building it in as the code is created.

We've already seen some really interesting examples of this in the work we're doing at ClickChain AI.

In one case, an agent started by writing a batch of tests up front and the gate stopped it — not because the tests were bad, but because the agent wasn't following the discipline of test-driven development. It had jumped ahead instead of letting one failing test drive the next behavior. So it had to go back: one failing test, make it pass, then the next.

That's governance.

In another case, we introduced a behavioral defect and the tests still passed. That's exactly the kind of thing I want to know. And the response wasn't simply, "Okay, add another test." We learned from it and changed the rule. That's what I want governance to do. Adeel wrote up both of those incidents in detail in the gate that blocked our own agent, including the numbers.

Expect. Build. Challenge. Verify. Learn. Continue.

Make the policy run

For me, mutation testing is a really good example of a much larger idea. I don't want software governance to become another checklist, another approval meeting, or another gate where somebody says yes or no based largely on judgment. Where we can, I want the governance to be executable and evidence-based.

Humans still decide what good looks like and what matters. They still make the judgments around architecture, intent, business risk, and context. But if there are things we can prove with the software itself, then prove them.

Make the policy run.

And whether the code is written by a developer, an AI agent, or some combination of the two, I keep coming back to the same question.

If the code is wrong, will your tests know it?