Warnings misuse

[webknjaz]

Summary

I've faced something that I believe is improper use of the warnings mechanism in Python.
Warnings are supposed to signal about something deprecated or called incorrectly. However, I see that it's being used to duplicate the reported information during regular operation (emitting a warning in addition to printing out the same and setting the test session outcome).

So in this instance, the end-user uses pytest-cov correctly, but still gets warnings.

I tend to surface warnings as errors so they're caught early, which causes it to show up as an internal pytest error. I'm convinced this shouldn't be happening.

Expected vs actual result

When I'm running python -Werror -Im pytest -- some/test.py, while having fail_under = 100, this causes a traceback.

Since it just runs a subset of tests, it's expected that coverage wouldn't be at 100%. And I'd expect it to mark the test session as failed.
What I didn't expect, though, was that it additionally issues a warning, which -Werror turns into an error resulting in a traceback being printed out.

Reproducer

Versions

$ .tox/py/bin/pip show pytest-cov
Name: pytest-cov
Version: 6.0.0
Summary: Pytest plugin for measuring coverage.
Home-page: https://github.com/pytest-dev/pytest-cov
Author: Marc Schlaich
Author-email: marc.schlaich@gmail.com
License: MIT
Location: ~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages
Requires: coverage, pytest
Required-by: 

$ .tox/py/bin/pip show pytest    
Name: pytest
Version: 8.3.4
Summary: pytest: simple powerful testing with Python
Home-page: https://docs.pytest.org/en/latest/
Author: Holger Krekel, Bruno Oliveira, Ronny Pfannschmidt, Floris Bruynooghe, Brianna Laugher, Florian Bruhin, Others (See AUTHORS)
Author-email: 
License: MIT
Location: ~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages
Requires: iniconfig, packaging, pluggy
Required-by: pytest-cov, pytest-mock, pytest-xdist

$ .tox/py/bin/pip show coverage  
Name: coverage
Version: 7.6.10
Summary: Code coverage measurement for Python
Home-page: https://github.com/nedbat/coveragepy
Author: Ned Batchelder and 235 others
Author-email: ned@nedbatchelder.com
License: Apache-2.0
Location: ~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages
Requires: 
Required-by: covdefaults, pytest-cov

Config

The covdefaults plugin of coveragepy sets fail_under = 100

#.coveragerc
[run]
plugins =
  covdefaults

Code

This is the repo version without the workaround: https://github.com/antonbabenko/pre-commit-terraform/tree/4b0ad90

tox -qq -- tests/pytest/_cli_test.py::test_app_exit -qq will do:

$ tox -qq -- tests/pytest/_cli_test.py::test_app_exit -qq
.
ERROR: Coverage failure: total of 61 is less than fail-under=100
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages/pluggy/_callers.py", line 156, in _multicall
INTERNALERROR>     teardown[0].send(outcome)
INTERNALERROR>   File "~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages/pytest_cov/plugin.py", line 355, in pytest_runtestloop
INTERNALERROR>     warnings.warn(CovFailUnderWarning(message), stacklevel=1)
INTERNALERROR> pytest_cov.CovFailUnderWarning: Coverage failure: total of 61 is less than fail-under=100
INTERNALERROR> 
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR> 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages/_pytest/main.py", line 283, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages/_pytest/main.py", line 337, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages/pluggy/_callers.py", line 160, in _multicall
INTERNALERROR>     _warn_teardown_exception(hook_name, teardown[1], e)
INTERNALERROR>   File "~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages/pluggy/_callers.py", line 50, in _warn_teardown_exception
INTERNALERROR>     warnings.warn(PluggyTeardownRaisedWarning(msg), stacklevel=5)
INTERNALERROR> pluggy.PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
INTERNALERROR> Plugin: _cov, Hook: pytest_runtestloop
INTERNALERROR> CovFailUnderWarning: Coverage failure: total of 61 is less than fail-under=100
INTERNALERROR> For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning

py: exit 3 (0.30 seconds) ~/src/github/antonbabenko/pre-commit-terraform> .tox/py/bin/python -bb -E -s -I -Werror -m pytest --color=yes tests/pytest/_cli_test.py::test_app_exit -qq pid=2113830

A workaround is to pass -W 'ignore:Coverage failure::pytest_cov.plugin' but it's rather a hack.

[ionelmc]

In a way you are right, those warning got nothing to do with deprecations. But if you extend the "incorect call" definition to also include "incorect configuration" then they do qualify.

If I remove those warnings then user won't have the option to make warning fail on any of these situations:

• Unexpected use of --no-cov
• Use of dynamic_context in coverage config (the plugin provides a better implementation)
• Test passed but there's a report generation failure
• Fail under check failed <- the problem you reported.

Now that I think of it maybe we could just remove that last warning, let me check.

[ionelmc]

Would #683 be enough?

[webknjaz]

But if you extend the "incorect call" definition to also include "incorect configuration" then they do qualify.

Agreed. I don't have a problem with warnings for things that should be fixed by the end-user. Just not the ones that are caused by legitimate configuration.