Code Coverage: Useful Metric or Trap?


Most development teams track code coverage. Many set a coverage threshold - 80%, 90% - that must pass for the build to succeed. The intention is sound: more covered code means more tested code means fewer bugs. The outcome is often the opposite: teams optimize for the metric rather than the property it’s supposed to measure, producing high coverage numbers and brittle test suites.

Understanding what coverage actually tells you - and what it doesn’t - makes it useful instead of counterproductive.

What Coverage Measures

Code coverage measures which lines, branches, or paths in your code are executed when the test suite runs. The common types:

Line coverage: percentage of lines executed. 100 / 150 lines executed = 67%.

Branch coverage: percentage of code branches taken. For an if/else, both the if branch and the else branch must be executed to cover both. A test that only exercises the happy path has low branch coverage even if it has high line coverage.

Path coverage: every unique path through a function. This grows exponentially with nesting and is rarely practical to measure at 100%.

Most teams and tools report line coverage or a combination of line and branch coverage.

What Coverage Does Not Measure

Coverage tells you that a line was executed. It does not tell you that the line was tested.

def calculate_discount(price, user_tier):
    if user_tier == 'premium':
        return price * 0.8
    return price

def test_calculate_discount():
    result = calculate_discount(100, 'premium')
    assert result is not None  # just checks it didn't return None

This test achieves 100% coverage of calculate_discount. It does not test that the discount is correct (20% off), that the base case returns the original price, that the function handles zero prices, negative prices, or unusual user tiers.

The test runs the code. It does not verify the code’s behavior.

A test suite with 90% line coverage and shallow assertions is worse than a test suite with 70% coverage and meaningful assertions - the first creates false confidence while the second tells you honestly what it doesn’t cover.

Goodhart’s Law Applied to Coverage

Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure.

When teams are required to maintain 80% coverage, they write tests that achieve 80% coverage. The most common patterns:

Testing trivial code to inflate numbers. Adding tests for getters, setters, and one-line utility functions. These run quickly, add coverage, and test nothing that could realistically break.

Assertions that don’t actually test anything. The example above - assert result is not None - technically passes while asserting nothing meaningful.

Marking code as excluded. Many coverage tools allow lines to be excluded from coverage reporting. Under pressure to hit a threshold, developers exclude the hard-to-test code rather than testing it.

Deleting tests that test the wrong thing. Tests that are slow or brittle get deleted to keep the suite fast, and no equivalent test gets written.

All of these achieve the coverage number while degrading the actual reliability of the software.

Where Coverage Is Useful

Coverage has genuine uses when treated as an indicator rather than a target.

Finding untested code paths. A coverage report shows which parts of the codebase have no tests. This is useful for identifying risky areas and making informed decisions about where to invest testing effort.

Preventing regression. In a codebase with existing tests, requiring that coverage doesn’t drop ensures that new code isn’t added untested. This is a defensive use of coverage thresholds - not “achieve 90%” but “don’t go below current levels.”

Spotting dead code. Code that is never reached by any test is either dead code (can be deleted) or critical code that tests don’t reach (needs to be tested). Both are worth knowing about.

Measuring test completeness during development. When writing a new module, coverage helps verify that the test cases you wrote actually exercise the paths you think they exercise. It’s a development tool, not just a reporting tool.

Branch Coverage Is More Informative Than Line Coverage

Line coverage can be 100% while branch coverage is 50%:

def get_user_discount(user):
    if user.is_premium and user.account_age_days > 365:
        return 0.20
    return 0.0

A test that calls this with a premium user who has been a member for 400 days achieves 100% line coverage. Branch coverage shows the untested case: a premium user with fewer than 365 days, or a non-premium user. The is_premium and age > 365 condition has four combinations; one test exercises one of them.

If your tool reports both, branch coverage is the more informative number. If you’re only looking at line coverage, the percentage overstates your confidence.

What Good Test Coverage Looks Like

The alternative to coverage-as-target is testing-as-specification.

Tests that matter:

  • Cover the business-critical paths thoroughly
  • Verify behavior under the important edge cases (empty inputs, boundary values, failure conditions)
  • Test error paths - what happens when a dependency fails, when input is invalid, when a concurrent request interferes
  • Are readable enough that their failure message tells you what broke and why

Coverage that comes from these tests is high because the code is tested, not because the metric was targeted. A team that writes this way typically achieves good coverage numbers as a side effect.

The question to ask about any test is: if this test fails in CI, will it tell me something useful about the system? A test that fails only when the function is deleted is less useful than a test that fails when the function’s behavior changes incorrectly.

The Practical Recommendation

Use coverage to find gaps, not to set thresholds. Run coverage reports during development to identify code paths that have no tests. Review them and decide whether they need tests based on the risk, not the coverage number.

If your team uses a threshold, make it a ratchet (don’t let coverage drop) rather than a gate (must exceed X%). This prevents new untested code without creating pressure to write bad tests for existing code.

The metric is real and useful. The target is where it turns counterproductive. The best-tested codebases tend to have high coverage as a natural consequence of careful testing, not because someone required a number.



Read more