automating the subprocess measurement setup

[nedbat]

Originally reported by Tibor (Bitbucket: tibor_arpas, GitHub: Unknown)

---

Subprocesses in test suits are a common occurence. coverage.py has an execellent way to measure their coverage but the set-up instructions are quite scary: http://nedbatchelder.com/code/coverage/subprocess.html

Also in the time of tox, continuous integration, virtualizations and containerizations, the manual process is quite beside the point.

What could we do to completely automate the described process?

There is an article by Ned http://nedbatchelder.com/blog/201001/running_code_at_python_startup.html

and a suggestion from @geoffbache

#!python
# sitecustomize.py
import os, sys
currDir = os.path.dirname(__file__)
sys.path.remove(currDir)
del sys.modules["sitecustomize"]
try:
    import sitecustomize
finally:
    sys.path.insert(0, currDir)

That script fails on my machine (MacOS, Python 2.7) with

#!stacktrace

Traceback (most recent call last):
  File "/Users/tibor/.virtualenvs/tmon/bin/../lib/python2.7/site.py", line 703, in <module>
    main()
  File "/Users/tibor/.virtualenvs/tmon/bin/../lib/python2.7/site.py", line 694, in main
    execsitecustomize()
  File "/Users/tibor/.virtualenvs/tmon/bin/../lib/python2.7/site.py", line 548, in execsitecustomize
    import sitecustomize
  File "/Users/tibor/tmonworkspace/testmon/sitecustomize.py", line 8, in <module>
    sys.path.insert(0, currDir)
AttributeError: 'NoneType' object has no attribute 'path'

But I would hope that it's the best approach.

---

• Bitbucket: https://bitbucket.org/ned/coveragepy/issue/367

[nedbat]

Is this a solution? https://github.com/dougn/coverage_pth

[nedbat]

Original comment by Tibor (Bitbucket: tibor_arpas, GitHub: Unknown)

---

Nice approach, I guess. I'm still getting ImportError: No module named
'coverage' on setup.py of coverage (tox/pip somehow always installs the pth
file first and only then attempts to run python setup.py of coverage, which
fails because there is no coverage.py yet)

[nedbat]

Original comment by Tibor (Bitbucket: tibor_arpas, GitHub: Unknown)

---

The problem I had is only with --editable coveragepy. So yes, it's definitely a solution! Could you include it in coveragepy itself?

[nedbat]

Original comment by Tibor (Bitbucket: tibor_arpas, GitHub: Unknown)

---

There is similar solution here: https://github.com/schlamar/pytest-cov/blob/772f2e210610354a8ffddc0187a23913ae56ca4e/cov-core/setup.py

[nedbat]

Original comment by Buck Evan (Bitbucket: bukzor, GitHub: bukzor)

---

The cov-core solution doesn't uninstall whatsoever. In fact it causes the whole virtualenv to explode (ImportError) if you uninstall it.

The coverage-pth solution is better, but doesn't handle setup.py develop aka pip install -e.

The solution found in python-hunter handles all those cases, in addition to supporting easy-install.
https://github.com/ionelmc/python-hunter/blob/master/setup.py

We've recently adapted this into a package that fills the same problem space as coverage-pth, but is more reliable.

IMO this should be a default feature of coveragepy. While it will add one if-statement to all subsequent python startup time, that seems like a near-zero cost to me, and the additional use-cases supported are sizeable.

[glyph]

I've been using https://pypi.org/project/coverage_enable_subprocess/ recently and it works very nicely.

[glyph]

The set-up instructions are also somewhat meandering. Should I try my hand at fixing the docs? It's really just 4 steps:

1. Write an explicit .coveragerc
2. Include parallel = true in it
3. Set COVERAGE_PROCESS_START to its absolute path (probably {toxinidir}/.coveragerc
4. ensure it's imported at process startup (by installing the aforementioned https://pypi.org/project/coverage_enable_subprocess/ )

In my humble opinion the way to fix this to be simpler would then be to remove non-parallel mode from coverage entirely, and have it take care of steps 1 (possibly putting coveragerc defaults in a temp dir) 3 (os.environ[...] = ...), and 4 (just ship the .pth file as part of coverage.py) automatically for you.

[nedbat]

@glyph tell me more about how "removing non-parallel mode entirely" would work? Wouldn't that mean that simple runs would always generate a numbered .coverage.xxx.yyy file?

[glyph]

@glyph tell me more about how "removing non-parallel mode entirely" would work? Wouldn't that mean that simple runs would always generate a numbered .coverage.xxx.yyy file?

That's what I'm suggesting, yes. Is there any particular reason not to do this?

[nedbat]

I don't understand what would happen to "coverage combine". Today, "coverage report" (or html, xml, etc), only report on data in the .coverage data file. Parallel files have to be combined first in order to be reported.

If we say everything is parallel from now on, does that mean that everyone has to run "coverage combine" before reporting? Or do the reporting commands all implicitly combine first? Running coverage twice in a row would give two distinct data files, then implicit combining would combine them together? That seems wrong.

[glyph]

Running coverage twice in a row would give two distinct data files, then implicit combining would combine them together? That seems wrong.

It's quite possible that my perspective is badly skewed by the fact that this is already the world I live in. I don't have any projects with only a single test environment, and once you have 2, you want to be able to speed up runs with tox -p auto, so parallel coverage is de rigeur even if your test suite itself has zero parallelism.

But let me take it as a given that coverage run thing1; coverage run thing2; coverage report reporting only on thing2 is desirable behavior. How do you get this with parallel coverage too? One strategy that occurs to me is that we add a discrete notion of a "coverage group", and give it its own identity. A coverage group would be a UUID with an optional descriptive label, and a timestamp. (I confess that "group" might be a terrible name for this concept, but, just to have something to call it...)

$ coverage group new --label "lol"
2e843587-a60c-4fab-bfaf-dab910437bb0
$ coverage run --group 2e843587-a60c-4fab-bfaf-dab910437bb0
$ coverage report --group 2e843587-a60c-4fab-bfaf-dab910437bb0
report for 'lol' ...

Without --group, coverage run implicitly creates a new group, and coverage report reports on the group with the latest creation timestamp.

You could then have coverage group --list to give you a nicely presented list of timestamps, IDs, and labels. (Groups would presumably be files with a naming convention, like .covgroup.$UUID with the timstamp and label in a very simple format inside them.)

To run a test suite that might run some subprocesses, you could then do, say:

$ coverage spawn tox -p auto
| / - \ | etc

Like run, this implicitly makes a new group, then runs tox -p auto as a subprocess with the coverage group ID as part of its environment.

$ coverage report

This reports on the group created by spawn, since it is the group with the most recent timestamp in the current directory.

Another advantage of this approach is that you could run multiple tests and report on their combined coverage at the same time in the same project, and you wouldn't need any special locking; .coverage files could be named .coverage.$GROUP.$HOST.$PID and commands like report could combine just the relevant files.