Skip to content

Fail coverage based on absolute number of uncovered lines #815

Open
@jml

Description

@jml

Is your feature request related to a problem? Please describe.
When improving the test coverage of a legacy code base, I would like to have a ratchet based on the absolute number of uncovered lines.

There are two aspects to this:

  1. Absolute number of uncovered lines

I want test coverage because untested code is very likely to be buggy. The number of bugs is not a function of the percentage of uncovered code, but rather the amount of uncovered code. Saying I've got 98% coverage might impress my friends, but if that means I've got 20kLoC of untested code then it's somewhat meaningless.

More seriously, deleting tested code reduces the test coverage percentage even though coverage is not objectively worse. Working around this takes all sorts of hijinx that are unnecessary when using absolute numbers.

  1. Ratchet

This means I would like coverage to fail not only if the coverage has got worse but also if it has got better. Failing if it's worse is obvious—I don't want coverage to get worse. Failing if it's better means that contributors will have to update the number of uncovered lines in the build scripts to lock in the newer, lower number.

Neither of these are a problem with greenfields projects, because then I start with --fail-under=100 immediately, which means I don't need a ratchet, and I have an absolute number of uncovered lines: zero.

Describe the solution you'd like
A new option to coverage report called --num-uncovered-lines=N that would cause it to exit with status 2 if there were more than N uncovered lines, and exit with status 3 if there were fewer than N uncovered lines.

The error message should clearly indicate whether coverage got worse or better. In the case where it's better, it should emit the new number of uncovered lines so it is easy to update build machinery.

Describe alternatives you've considered
If the ratchet aspect is distasteful, having --max-uncovered-lines=N would also be very useful.

Additional context
Here's the script that I'm using for this right now: https://gist.github.com/jml/27d21c9322d112a3b9b5e94508c93928

I run it in CI with something like:

# The number of lines currently without test coverage.
# Do not approve a PR that increases this number.
UNCOVERED_LINES = 761

coverage run --branch -m pytest -q tests/
coverage report --show-missing --skip-covered
python scripts/missing_coverage_lines.py --allowed-missing=${UNCOVERED_LINES} .coverage

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions