Module code is executed twice when collecting doctests

[flying-sheep]

• a detailed description of the bug or problem you are having
• output of pip list from the virtual environment you are using
• pytest and operating system versions
• minimal example if possible

Bug description and minimal example

https://github.com/flying-sheep/pytest-doctest-import-twice

Versions

OS: Arch Linux

pip list --format=json --exclude-editable | from json | to md --pretty

name	version
iniconfig	2.0.0
packaging	23.1
pip	23.2.1
pluggy	1.2.0
pytest	7.4.0
setuptools	68.0.0
wheel	0.41.0

[The-Compiler]

Thanks for the detailed example! However, it seems to fail initially with:

        File "/tmp/pip-build-env-d1izs26m/overlay/lib/python3.11/site-packages/hatchling/metadata/core.py", line 508, in readme
          raise OSError(message)
      OSError: Readme file does not exist: README.md

after fixing that:

diff --git i/pyproject.toml w/pyproject.toml
index ecb2f55..c797e64 100644
--- i/pyproject.toml
+++ w/pyproject.toml
@@ -7,7 +7,7 @@ name = 'pytest-doctest-import-twice'
 version = '1.0'
 description = 'pytest reproducer for doctest import behavior'
 authors = [{ name = 'Philipp A.', email = 'flying-sheep@web.de' }]
-readme = 'README.md'
+readme = 'README.rst'
 license = 'GPL-3.0-or-later'
 requires-python = '>=3.11'
 dependencies = [

it looks like it also needs an export PYTHONPATH=. to work.

I ended up creating a virtualenv manually and using .venv/bin/pytest --import-mode=importlib --doctest-modules -o pythonpath=. to reproduce instead. Surprisingly, the example works fine with pytest 7.1.0, and I was able to bisect this to:

• Pass importmode to import_path in DoctestModule #10088

However, that looks like it just happens to expose this problem, and isn't actually the culprit itself.

[flying-sheep]

it looks like it also needs an export PYTHONPATH=. to work.

Or equivalent. hatch run … (in the README) creates a venv for you and installs your package in dev mode before running the specified script. As far as the import system is concerned, a dev install is just adding the parent directory to sys.path, only using a .pth file instead of an env variable.

If you do a regular install and run Python in safepath (-P) mode, the error is still reproducible:

$ hatch run python -Pm site
sys.path = [
    '/usr/lib/python311.zip',
    '/usr/lib/python3.11',
    '/usr/lib/python3.11/lib-dynload',
    '<venv>/lib/python3.11/site-packages',
]
[…]
$ hatch run python -Pm pytest
[…]
collected 1 item / 1 error
[…]

the same applies to doing a regular install and running the pytest binary (which is probably equivalent to safepath mode anyway)

$ hatch run pytest
[…]
collected 1 item / 1 error
[…]

[flying-sheep]

The takeaway of all this is that this is not an issue of setting the PYTHONPATH wrong:

It’s an issue with the way pytest’s imports things by path. This code path should be hit the second time:

pytest/src/_pytest/pathlib.py

Lines 524 to 527 in 556e075

	if mode is ImportMode.importlib:
	module_name = module_name_from_path(path, root)
	with contextlib.suppress(KeyError):
	return sys.modules[module_name]

/edit: wait a second, that code path doesn’t exist in my local copy

[The-Compiler]

yes, I'm not saying this is caused by PYTHONPATH. But your example needs the PYTHONPATH set to run, even when using hatch run, at least on my system:

$ hatch run pytest
============================= test session starts ==============================
platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0
rootdir: /home/florian/tmp/11306/pytest-doctest-import-twice
configfile: pyproject.toml
collected 0 items / 3 errors                                                   

==================================== ERRORS ====================================
___________ ERROR collecting pytest_doctest_import_twice/__init__.py ___________
../../../.local/share/hatch/env/virtual/pytest-doctest-import-twice/fxat3dFr/pytest-doctest-import-twice/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
../../../.local/share/hatch/env/virtual/pytest-doctest-import-twice/fxat3dFr/pytest-doctest-import-twice/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
../../../.local/share/hatch/env/virtual/pytest-doctest-import-twice/fxat3dFr/pytest-doctest-import-twice/lib/python3.11/site-packages/_pytest/doctest.py:547: in collect
    module = import_path(
../../../.local/share/hatch/env/virtual/pytest-doctest-import-twice/fxat3dFr/pytest-doctest-import-twice/lib/python3.11/site-packages/_pytest/pathlib.py:538: in import_path
    spec.loader.exec_module(mod)  # type: ignore[union-attr]
<frozen importlib._bootstrap_external>:940: in exec_module
    ???
<frozen importlib._bootstrap>:241: in _call_with_frames_removed
    ???
pytest_doctest_import_twice/__init__.py:1: in <module>
    from .singleton import MyVindictiveSingleton
E   ModuleNotFoundError: No module named 'pytest_doctest_import_twice'
_____________________ ERROR collecting tests/test_basic.py _____________________
../../../.local/share/hatch/env/virtual/pytest-doctest-import-twice/fxat3dFr/pytest-doctest-import-twice/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
../../../.local/share/hatch/env/virtual/pytest-doctest-import-twice/fxat3dFr/pytest-doctest-import-twice/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
../../../.local/share/hatch/env/virtual/pytest-doctest-import-twice/fxat3dFr/pytest-doctest-import-twice/lib/python3.11/site-packages/_pytest/doctest.py:547: in collect
    module = import_path(
../../../.local/share/hatch/env/virtual/pytest-doctest-import-twice/fxat3dFr/pytest-doctest-import-twice/lib/python3.11/site-packages/_pytest/pathlib.py:538: in import_path
    spec.loader.exec_module(mod)  # type: ignore[union-attr]
../../../.local/share/hatch/env/virtual/pytest-doctest-import-twice/fxat3dFr/pytest-doctest-import-twice/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
tests/test_basic.py:1: in <module>
    from pytest_doctest_import_twice import instance
E   ImportError: cannot import name 'instance' from 'pytest_doctest_import_twice' (unknown location)
_____________________ ERROR collecting tests/test_basic.py _____________________
ImportError while importing test module '/home/florian/tmp/11306/pytest-doctest-import-twice/tests/test_basic.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_basic.py:1: in <module>
    from pytest_doctest_import_twice import instance
E   ImportError: cannot import name 'instance' from 'pytest_doctest_import_twice' (unknown location)
=========================== short test summary info ============================
ERROR pytest_doctest_import_twice/__init__.py - ModuleNotFoundError: No module named 'pytest_doctest_import_twice'
ERROR tests/test_basic.py - ImportError: cannot import name 'instance' from 'pytest_doctest_import_twic...
ERROR tests/test_basic.py
!!!!!!!!!!!!!!!!!!! Interrupted: 3 errors during collection !!!!!!!!!!!!!!!!!!!!
============================== 3 errors in 0.10s ===============================

[flying-sheep]

That’s shouldn’t be the case. As said, hatch should just make a venv, dev-install the project into it, and then run the script.

Are you using an old hatch version (≠1.7.0)? Maybe try manually creating a venv?

#!/bin/bash
test -d .venv || python -m virtualenv .venv
source .venv/bin/activate
pip install -e .  # or pip install .
pytest

It shouldn’t matter (as said, both PYTHONPATH and regular install reproduce the bug) but it’s worrying that something on our systems behaves so differently.

You can also try to recreate the env using hatch env prune and then hatch run pytest.

[flying-sheep]

wait a second, the code path I saw above has not been released. It’s part of #11148

I updated the test code to use pytest master and add some context about what pytest is doing.

Seems like that code didn’t quite fix the problem, I see:

E   module_name: 'pytest_doctest_import_twice.__init__'
E  […]
E   -----------------------------------
E   module_name: 'pytest_doctest_import_twice.__init__'
E  […]

shouldn’t that cause the return sys.modules[module_name] line to succeed one of the two times?

[flying-sheep]

@nicoddemus any idea how it’s possible that the line throws a KeyError twice with the same argument?

[nicoddemus]

Hi @flying-sheep sorry not a glance, I think this might need debugging to understand what is happening.

[flying-sheep]

I hope I provided a good enough reproducer for that! Glad it’s on your radar.

[nicoddemus]

Oh sorry I was not clear, I did not mean I will be working on this soon, I'm on vacation at the moment.

[flying-sheep]

Haha no worries, enjoy!