Skip to content

Proper tracking of conditional "partial" lines #1962

Open
@kingbuzzman

Description

@kingbuzzman

Is your feature request related to a problem? Please describe.

Currently, when using code coverage with complex boolean expressions in conditional statements (if, elif), the reporting lacks sufficient granularity.

For example, consider a condition like:

if a or b or c or d:
    # code block A
elif e or f:
    # code block B
else:
    # code block C

Standard branch coverage often marks the entire line if a or b or c or d: as "covered" if the condition evaluates to True, regardless of which specific sub-expression (a, b, c, or d) caused it to be true due to short-circuiting. This means we might execute code block A because b was True, but never actually test the cases where a, c, or d would have individually caused the condition to be True.

This makes it difficult to ensure that all logical paths within a complex boolean expression have been exercised by the test suite, particularly in large or legacy codebases.

Describe the solution you'd like

I would like the coverage tool to provide more detailed insight into the evaluation of complex boolean expressions within conditionals. Specifically, it would be beneficial to know which individual sub-expressions within a compound condition were evaluated and which ones were decisive in determining the outcome (True or False) of the condition.

Ideally, the coverage report could indicate if a boolean expression was only "partially" covered, meaning not all potential evaluation paths or short-circuiting scenarios within that specific condition were tested.

Possible ways to report this could include:

  1. Annotated Source View: Highlighting or annotating parts of the boolean expression that were evaluated or determined the outcome during a test run.

    # Example annotation showing 'b' was the decisive factor:
    if a # F, evaluated # or b # T, decisive # or c # Not evaluated # or d # Not evaluated #:
        ...

    Or perhaps marking the line as "partial" if not all sub-expressions were evaluated across all test runs hitting that line.

  2. Detailed Report/Table: A separate report or section detailing complex conditions and which specific components within them were triggered by which test cases.

Let's consider the expression if a or (b and (c or d)):.

Test Case Section 1 (a) Evaluation Section 2 (c or d) Evaluation (if reached) Section 3 (b and (c or d)) Evaluation Overall Condition Outcome
test_a_is_true True Short-Circuited Short-Circuited True
test_a_false_b_true_c_true False True (because c is True) True (because b & Sec 2 are True) True
test_a_false_b_true_d_true False True (because d is True) True (because b & Sec 2 are True) True
test_a_false_b_true_c_d_false False False False (because Sec 2 is False) False
test_a_false_b_false False Short-Circuited False (because b is False) False

Please note that I'm only advocating for this to be shown on or since the current solution works amazing for and.

Describe alternatives you've considered

The primary alternative is to refactor the complex boolean expressions into simpler conditions, potentially breaking them down into multiple lines, intermediate variables, or helper functions.

However, this presents a catch-22 situation: refactoring risky legacy code requires high confidence in the existing test suite to ensure no behavior is changed unintentionally. Without detailed condition coverage information, it is difficult to know if the current tests are sufficient to cover all the nuances of the existing complex logic before attempting to refactor it. This feature would provide the necessary insight to confidently identify testing gaps and improve coverage prior to or alongside refactoring efforts.


Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions