The Three-Score Problem66.67% / 91.04% / 100%

Mutation testing introduces small, syntactic changes — mutants — into the source code, then runs the test suite against each one. If the suite catches the change, the mutant is "killed." If the suite still passes, the mutant "survives." The mutation score is the ratio of killed to total mutants. Survival can mean two distinct things: the tests fail to verify the affected behaviour, or the mutation produced no observable change at all. Distinguishing those two cases is most of the work of reading mutation reports honestly, and most of what the rest of this note is about.

The controversy starts at the score. Three different ratios can be computed from the same mutmut run, and each is in active use across the industry. None of them is mathematically wrong.

Raw kill rate counts only unambiguous kills. It is the defensible floor. The number cannot be inflated by methodology choices, which is exactly its value: any reviewer of any mutation report can compute it without trusting the reporter's judgement.

Detected rate counts mutmut's suspicious cohort as kills. mutmut flags a mutant as suspicious based purely on execution time: when the test run for the mutated module takes significantly longer than the unmutated baseline but doesn't hit the hard timeout. The intuition is that a slow run is probably one that's struggling to fail. The heuristic is operational, not semantic. Most mutation tooling treats suspicious as detected by default. The convention silently transfers an ambiguity from the test suite to the reporting framework. Reading detected as a coverage figure requires the reporter to have inspected the suspicious cohort and confirmed that none of them are genuine misses dressed up as slow tests.

Adjusted rate excludes the surviving mutants from the denominator on the grounds that they are equivalent — semantically identical to the original code under all execution paths the system reaches. This is the strongest claim the three numbers can carry, because it removes data rather than adding it. If even one survivor is misclassified as equivalent, the adjusted rate is overstating coverage.

Each successive score therefore depends on a different amount of human judgement. Raw is automatic. Detected requires a defence of the runner's tooling decisions. Adjusted requires a defence of the reporter's own. Mutation reports in the wild routinely quote the highest available number with no methodology disclosed. The next two sections walk through what that disclosure should actually contain, using the surviving mutants from this dataset as the worked example.

Open the surviving-mutants list in MutationTelemetry.json. There are twenty-five of them. Twenty-one follow one pattern; four follow others. The first one:

The mutation is mutmut's string-replacement strategy. It found a string literal — "0.00" — and inserted its XX…XX markers. The resulting code, Decimal("XX0.00XX"), is not a valid Decimal constructor argument. If executed, it raises decimal.InvalidOperation.

The line is a dataclass field default. In Python, dataclass defaults are evaluated when the class is defined, not when an instance is created. The mutated default raises immediately on import. The module never loads. Every test fails. A mutation runner that detects import-level test failures should record this as a kill.

In the run that produced this telemetry, mutmut v2 didn't. The mechanism is mundane: when a mutated module fails to import, pytest exits with code 2 (collection error). Older versions of mutmut register a kill on exit code 1 specifically and default everything else, including code 2, to "survived." The mutated module never imported, no tests ran, and the runner wrote the result as a survivor. This is a documented bug in the runner's exit-code parsing, not a property of the test suite under examination.

In the mutation testing literature these are called incompetent or stillborn mutants — mutants that fail to produce a runnable program. PIT, the dominant mutation testing tool in the Java ecosystem, isolates them explicitly as RUN_ERROR or MEMORY_ERROR and excludes them from the mutation score by default. mutmut, working at the Python AST level, generates them by design and then misclassifies them at the parsing layer, which is how twenty-one of them ended up in this telemetry's "survived" column. None of this is a property of the code under test.

This pattern accounts for twenty-one of the twenty-five surviving mutants in the dataset. Sequential dataclass field defaults across the module: balance, frozenAmt, drBalance, crBalance, drTransactionAmt, and on through line 76. None of them survives in any meaningful sense. They are stillborn mutants the runner mislabelled.

The remaining four survivors are different in kind. Each is a genuine equivalent with a reachability argument that can be checked by reading the surrounding code.

These four are real equivalents — each with a one-sentence reachability argument that can be verified by reading the surrounding code. The other twenty-one are not equivalents at all. They are stillborn mutants on a known mutmut v2 parsing bug.

The "all 25 are equivalent" classification that produces a 100% adjusted score is correct numerically. It is also wrong informationally. Without the classification, a team reading "100% adjusted" cannot know that 84% of the exemptions are tooling errors and only 16% are genuine reachability proofs. Both readings produce 100%. Only one tells you what the test suite actually proves.

This is the loose-versus-strict distinction in operational terms. The loose reading: equivalent. The strict reading: twenty-one Class A stillborn mutants on a known mutmut v2 parsing bug; four Class B genuine equivalents with written reachability proofs. The framework that produces the second statement is the next section.

The framework below distinguishes the three things a "surviving" mutant can actually represent. Two of the categories are well-established in the mutation testing literature: Class A is what Offutt and others call incompetent or stillborn mutants; Class B is the much-discussed Equivalent Mutant Problem, formally undecidable in the general case, which is why the framework demands written justification per mutant rather than algorithmic detection. The contribution here is operational rather than theoretical: every survivor is classified A, B, or C, with written justification per mutant, included as an appendix to any report that cites an adjusted score.

The framework deliberately ignores cases that don't map cleanly to behaviour: mutations on log strings, debug paths, or __repr__ output. These are real survivors but extending the test suite to assert on them is itself an anti-pattern. Where they appear in volume on a real codebase, the right response is a separate exclusion list maintained by the team, not a fourth category that pretends every line of code is a coverage candidate.

The order of the columns matters more than the categories themselves. Most mutation reports are produced by computing the score first, then writing the methodology to fit. The framework above inverts this: classification rules are committed to before the survivors are inspected, and the score is whatever falls out.

Applied to the dataset in §1, the framework treats twenty-one survivors as Class A — stillborn mutants that should never have entered the denominator at all, since they fail to produce a runnable program. The remaining four are Class B: genuine equivalents, excluded with per-mutant reachability justification. The adjusted score is still 100%, but for two distinct reasons: 21 mutants that don't count, and 4 that count but cancel. The number is unchanged. The meaning of the number is not.