Branch coverage of try/except

[bensimner]

Is your feature request related to a problem? Please describe.
Consider the following code:

def f(x):
  try:
    y = 1/x
  except ZeroDivisionError:
    y = 0
  return y


def test_exception_branch_coverage():
  f(0)


test_exception_branch_coverage()

Here f(0) clearly misses a path: the case where 1/x did not raise an exception isn't covered. However Coverage.py (version 4.5.4) reports 100% statement and branch coverage.

Describe the solution you'd like
A try/except is like a branch. Consider the following:

def g(x):
  try:
    y = 1/x
  except ZeroDivisionError:
    y = 0
  else:
    pass  # this line wasn't executed, but coverage doesn't report it as missed
  return y

g(0)

The else represents a branch target that was never taken. Coverage could know about the branch intrinsic in the try/except/else itself: whether an exception was raised or not.

Describe alternatives you've considered
The else block does not always represent a missed branch:

def h(x):
  try:
    return 1/x
  except ZeroDivisionError:
    return 0
  else:
    pass  # never executed, but this does not imply a missed branch ...

h(3)
h(0)

Instead of just looking at whether the whole try threw an exception, one could consider each line individually:

def i(x):
  try:
    a = 1/(x+1)
    b = 1/(x+2)
    return a + b
  except ZeroDivisionError:
    return 0

i(0)
i(-1)

At first, it appears that all branches were covered by the above. i(0) executes the entire try block and throws no exception. i(-1) throws an exception and executes the except block.
However, both a = ... and b = ... could raise exceptions and only one of them was tested.

Keeping track of each line like this is tracking too much: in real code, not every line can raise interesting exceptions and those that don't shouldn't be considered missed branches:

def j(x):
  try: 
    y = x[0]
    return y + 1  # this line never raised an IndexError, but that's ok
  except IndexError:
    return None

j([])
j([1])

The return y + 1 line cannot raise an IndexError and therefore isn't a missed branch.

Additional context
It's not clear whether coverage tools should consider these "branches", or whether they should even report this kind of thing. One would expect that most of the time, statement coverage will catch a try/except that misses one of these cases, and it's only in the situation where there's a one-line try where later code is not conditional on the exception happening/not happening.

Maybe if added behaviours like this should be hidden behind flags for those that want more pedantic coverage metrics.

With thanks to @edk0 who originally pointed this out and for comments.

[edk0]

I've been using the following hack to experiment with this feature without having to implement it:

import sys

try_stack = []

with open(sys.argv[1], 'r') as f:
    lines = [*f, '']

sys.stdout = open(sys.argv[1], 'w')

last_block_stmt = None
jump = False

for i, line in enumerate(lines):
    line = line.rstrip('\n')
    bare_line = line.lstrip(' ')
    indent_level = len(line) - len(bare_line)
    append = []
    if ':' in bare_line:
        block_stmt = bare_line.partition(':')[0].split(' ')[0]
    else:
        block_stmt = None

    if block_stmt == 'try':
        jump = False
    elif block_stmt == 'except':
        if not try_stack or try_stack[-1] < indent_level:
            try_stack.append(indent_level)

    if bare_line and block_stmt != 'except':
        # we might need to close things
        while try_stack and indent_level <= try_stack[-1]:
            try_level = try_stack.pop(-1)
            if jump:
                continue
            if block_stmt == 'else' and indent_level == try_level:
                append.append(f"{' ' * indent_level}    assert True")
                break
            print(f"{' ' * try_level}else:")
            print(f"{' ' * try_level}    assert True")
        if block_stmt is None and last_block_stmt == 'try':
            jump = bare_line.split(' ')[0] in ('return', 'continue', 'break')

    if i < len(lines) - 1:
        print(line)
    for a in append:
        print(a)
    last_block_stmt = block_stmt or last_block_stmt

While obviously far from perfect, so far it's at least told me that the feature as requested doesn't have false positives on my code.

[nedbat]

Thanks for the thought-provoking issue, and @edk0 for the impressive hack to try it out :)

I think just as branch coverage can shine light on the "missing implicit else" in if-statements, we can try doing the same for try-blocks. I have to get 5.0 out the door before I can experiment with it, though.

[postmasters]

I have a case where the lines are reported as "missing" but are indeed covered, with branching on. Line 11 to 14 is from the if statement to print('3'). As you can see, all 1, 2, and 3 were printed.

tests/module.py

class Creds:

  def __init__(self, store):
    self.store = store

  def func(self):
    try:
      1/0
    except:
      print('1')
      if self.store:
        print('2')
        self.store.unlock()
      print('3')
      raise

tests/test_coverage.py

from . import module

import mock
import unittest

class TestCoverage(unittest.TestCase):

  def test_coverage(self):
    store = mock.Mock()
    creds = module.Creds(store)
    with self.assertRaises(Exception):
      creds.func()

py.test tests/test_coverage.py --cov=tests.module -s --cov-branch

tests/test_coverage.py 1
2
3
.

----------- coverage: platform linux, python 3.8.5-final-0 -----------
Name              Stmts   Miss Branch BrPart  Cover   Missing
-------------------------------------------------------------
tests/module.py      13      0      2      1    93%   11->14

py.test tests/test_coverage.py --cov=tests.module -s --cov-branch

tests/test_coverage.py 1
2
3
.

----------- coverage: platform linux, python 3.8.5-final-0 -----------
Name              Stmts   Miss  Cover   Missing
-----------------------------------------------
tests/module.py      13      0   100%