Pytest run fails under Python 3.10.0-beta.2 on Windows

[sergeyklay]

Traceback (most recent call last):
  ...
  File "d:\a\gstore\gstore\.venv\lib\site-packages\_pytest\capture.py", line 90, in _readline_workaround
    import readline  # noqa: F401
  File "d:\a\gstore\gstore\.venv\lib\site-packages\readline.py", line 34, in <module>
    rl = Readline()
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pyreadline\rlmain.py", line 422, in __init__
    BaseReadline.__init__(self)
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pyreadline\rlmain.py", line 62, in __init__
    mode.init_editing_mode(None)
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pyreadline\modes\emacs.py", line 633, in init_editing_mode
    self._bind_key('space',       self.self_insert)
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pyreadline\modes\basemode.py", line 162, in _bind_key
    if not callable(func):
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pyreadline\py3k_compat.py", line 8, in callable
    return isinstance(x, collections.Callable)
AttributeError: module 'collections' has no attribute 'Callable'

full traceback

Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.10.0-beta.2\x64\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\hostedtoolcache\windows\Python\3.10.0-beta.2\x64\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "D:\a\gstore\gstore\.venv\Scripts\py.test.exe\__main__.py", line 7, in <module>
  File "d:\a\gstore\gstore\.venv\lib\site-packages\_pytest\config\__init__.py", line 185, in console_main
    code = main()
  File "d:\a\gstore\gstore\.venv\lib\site-packages\_pytest\config\__init__.py", line 143, in main
    config = _prepareconfig(args, plugins)
  File "d:\a\gstore\gstore\.venv\lib\site-packages\_pytest\config\__init__.py", line 318, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pluggy\hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pluggy\callers.py", line 203, in _multicall
    gen.send(outcome)
  File "d:\a\gstore\gstore\.venv\lib\site-packages\_pytest\helpconfig.py", line 100, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pluggy\callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pluggy\callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "d:\a\gstore\gstore\.venv\lib\site-packages\_pytest\config\__init__.py", line 1003, in pytest_cmdline_parse
    self.parse(args)
  File "d:\a\gstore\gstore\.venv\lib\site-packages\_pytest\config\__init__.py", line 1283, in parse
    self._preparse(args, addopts=addopts)
  File "d:\a\gstore\gstore\.venv\lib\site-packages\_pytest\config\__init__.py", line 1191, in _preparse
    self.hook.pytest_load_initial_conftests(
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pluggy\hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pluggy\manager.py", line 84, in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pluggy\callers.py", line 208, in _multicall
    return outcome.get_result()
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pluggy\callers.py", line 80, in get_result
    raise ex[1].with_traceback(ex[2])
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pluggy\callers.py", line 182, in _multicall
    next(gen)  # first yield
  File "d:\a\gstore\gstore\.venv\lib\site-packages\_pytest\capture.py", line 157, in pytest_load_initial_conftests
    _readline_workaround()
  File "d:\a\gstore\gstore\.venv\lib\site-packages\_pytest\capture.py", line 90, in _readline_workaround
    import readline  # noqa: F401
  File "d:\a\gstore\gstore\.venv\lib\site-packages\readline.py", line 34, in <module>
    rl = Readline()
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pyreadline\rlmain.py", line 422, in __init__
    BaseReadline.__init__(self)
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pyreadline\rlmain.py", line 62, in __init__
    mode.init_editing_mode(None)
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pyreadline\modes\emacs.py", line 633, in init_editing_mode
    self._bind_key('space',       self.self_insert)
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pyreadline\modes\basemode.py", line 162, in _bind_key
    if not callable(func):
  File "d:\a\gstore\gstore\.venv\lib\site-packages\pyreadline\py3k_compat.py", line 8, in callable
    return isinstance(x, collections.Callable)
AttributeError: module 'collections' has no attribute 'Callable'

I discovered this in the GitHub Actions, with Pytest 6.2.4 on Microsoft Windows Server 2019 and 3.10.0-beta.2 from actions/setup-python@v2.2.2. Unfortunately I don't have a smaller reproducing example than that. Please let me know if I can provide more info.

---

P.S. Everything is OK with the same Python and pytest versions on macOs and Linux.

[RonnyPfannschmidt]

at first glance it seems a very outdated library that no longer works (and messes with readline) is installed

https://pypi.org/project/pyreadline/2.1/ was last released in 2015 and doesn't correctly work on modern python3

@nicoddemus the pytest readline workaround for windows mentions this library specifically, it seems like we need to blacklist it on modern python+pytest

its really unfortunate if support libs go unmaintained that way

[domdfcoding]

This has been reported to pyreadline as pyreadline/pyreadline#65

@sergeyklay Looks like you depend on humanfriendly for your tests? There is an open issue for this there as well: xolox/python-humanfriendly#44

To unblock my own tests I've been using the https://github.com/osrf/pyreadline fork. If you want a pre-built wheel there's one in https://github.com/domdfcoding/3.10-wheels

[sergeyklay]

@domdfcoding As a far I can see there is neither humanfriendly nor pyreadline in the dependency tree:

$ pipdeptree --python .tox/py310/bin/python 
coverage==5.5
gstore==0.5.0
  - gitpython [required: >=3.0.6, installed: 3.1.17]
    - gitdb [required: >=4.0.1,<5, installed: 4.0.7]
      - smmap [required: >=3.0.1,<5, installed: 4.0.0]
  - pygithub [required: >=1.55, installed: 1.55]
    - deprecated [required: Any, installed: 1.2.12]
      - wrapt [required: >=1.10,<2, installed: 1.12.1]
    - pyjwt [required: >=2.0, installed: 2.1.0]
    - pynacl [required: >=1.4.0, installed: 1.4.0]
      - cffi [required: >=1.4.1, installed: 1.14.5]
        - pycparser [required: Any, installed: 2.20]
      - six [required: Any, installed: 1.16.0]
    - requests [required: >=2.14.0, installed: 2.25.1]
      - certifi [required: >=2017.4.17, installed: 2021.5.30]
      - chardet [required: >=3.0.2,<5, installed: 4.0.0]
      - idna [required: >=2.5,<3, installed: 2.10]
      - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.5]
pip==21.1.2
pytest==6.2.4
  - attrs [required: >=19.2.0, installed: 21.2.0]
  - iniconfig [required: Any, installed: 1.1.1]
  - packaging [required: Any, installed: 20.9]
    - pyparsing [required: >=2.0.2, installed: 2.4.7]
  - pluggy [required: >=0.12,<1.0.0a1, installed: 0.13.1]
  - py [required: >=1.8.2, installed: 1.10.0]
  - toml [required: Any, installed: 0.10.2]
setuptools==57.0.0
wheel==0.36.2

However, I'm not a Windows user, and this output was received in macOs. Probably something is hidden here:

pytest/src/_pytest/capture.py

Lines 88 to 92 in 8300b26

	if sys.platform.startswith("win32"):
	try:
	import readline # noqa: F401
	except ImportError:
	pass

As you could see above, in the stacktrace I provided pytest calls readline and readline calls pyreadline:
pytest -> readline -> pyreadline

The last commit to pyreadline was on 14 Jul 2015 and it looks like it does not support Python 3.10.

[asottile]

@sergeyklay dependencies are platform-dependent, doing a deptree on macos does not tell you what will be installed on windows

[sergeyklay]

I have no idea what has changed, but now the build and test are passing! I just waited a few days and tried again. I suspect that due to the fact that I rewrote some [conf]tests, some functionality that requires pyreadline is no longer called. However, unfortunately I cannot determine what exactly

Btw, I managed to get the full dependency tree in GitHub Actions using Python 3.10 on Windows, and agin, there is neither humanfriendly nor pyreadline ¯_(ツ)_/¯ :

furo==2020.12.30b24
  - beautifulsoup4 [required: Any, installed: 4.9.3]
    - soupsieve [required: >1.2, installed: 2.2.1]
  - sphinx [required: ~=3.0, installed: 3.5.4]
    - alabaster [required: >=0.7,<0.8, installed: 0.7.12]
    - babel [required: >=1.3, installed: 2.9.1]
      - pytz [required: >=2015.7, installed: 2021.1]
    - colorama [required: >=0.3.5, installed: 0.4.4]
    - docutils [required: >=0.12,<0.17, installed: 0.16]
    - imagesize [required: Any, installed: 1.2.0]
    - Jinja2 [required: >=2.3, installed: 3.0.1]
      - MarkupSafe [required: >=2.0, installed: 2.0.1]
    - packaging [required: Any, installed: 20.9]
      - pyparsing [required: >=2.0.2, installed: 2.4.7]
    - Pygments [required: >=2.0, installed: 2.9.0]
    - requests [required: >=2.5.0, installed: 2.25.1]
      - certifi [required: >=2017.4.17, installed: 2021.5.30]
      - chardet [required: >=3.0.2,<5, installed: 4.0.0]
      - idna [required: >=2.5,<3, installed: 2.10]
      - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.5]
    - setuptools [required: Any, installed: 56.0.0]
    - snowballstemmer [required: >=1.1, installed: 2.1.0]
    - sphinxcontrib-applehelp [required: Any, installed: 1.0.2]
    - sphinxcontrib-devhelp [required: Any, installed: 1.0.2]
    - sphinxcontrib-htmlhelp [required: Any, installed: 2.0.0]
    - sphinxcontrib-jsmath [required: Any, installed: 1.0.1]
    - sphinxcontrib-qthelp [required: Any, installed: 1.0.3]
    - sphinxcontrib-serializinghtml [required: Any, installed: 1.1.5]
gstore==0.5.0
  - gitpython [required: >=3.0.6, installed: 3.1.17]
    - gitdb [required: >=4.0.1,<5, installed: 4.0.7]
      - smmap [required: >=3.0.1,<5, installed: 4.0.0]
  - pygithub [required: >=1.55, installed: 1.55]
    - deprecated [required: Any, installed: 1.2.12]
      - wrapt [required: >=1.10,<2, installed: 1.12.1]
    - pyjwt [required: >=2.0, installed: 2.1.0]
    - pynacl [required: >=1.4.0, installed: 1.4.0]
      - cffi [required: >=1.4.1, installed: 1.14.5]
        - pycparser [required: Any, installed: 2.20]
      - six [required: Any, installed: 1.16.0]
    - requests [required: >=2.14.0, installed: 2.25.1]
      - certifi [required: >=2017.4.17, installed: 2021.5.30]
      - chardet [required: >=3.0.2,<5, installed: 4.0.0]
      - idna [required: >=2.5,<3, installed: 2.10]
      - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.5]
pipdeptree==2.0.0
  - pip [required: >=6.0.0, installed: 21.1.2]
pytest==6.2.4
  - atomicwrites [required: >=1.0, installed: 1.4.0]
  - attrs [required: >=19.2.0, installed: 21.2.0]
  - colorama [required: Any, installed: 0.4.4]
  - iniconfig [required: Any, installed: 1.1.1]
  - packaging [required: Any, installed: 20.9]
    - pyparsing [required: >=2.0.2, installed: 2.4.7]
  - pluggy [required: >=0.12,<1.0.0a1, installed: 0.13.1]
  - py [required: >=1.8.2, installed: 1.10.0]
  - toml [required: Any, installed: 0.10.2]
tox-gh-actions==2.5.0
  - tox [required: >=3.12, installed: 3.23.1]
    - colorama [required: >=0.4.1, installed: 0.4.4]
    - filelock [required: >=3.0.0, installed: 3.0.12]
    - packaging [required: >=14, installed: 20.9]
      - pyparsing [required: >=2.0.2, installed: 2.4.7]
    - pluggy [required: >=0.12.0, installed: 0.13.1]
    - py [required: >=1.4.17, installed: 1.10.0]
    - six [required: >=1.14.0, installed: 1.16.0]
    - toml [required: >=0.9.4, installed: 0.10.2]
    - virtualenv [required: >=16.0.0,!=20.0.7,!=20.0.6,!=20.0.5,!=20.0.4,!=20.0.3,!=20.0.2,!=20.0.1,!=20.0.0, installed: 20.4.7]
      - appdirs [required: >=1.4.3,<2, installed: 1.4.4]
      - distlib [required: >=0.3.1,<1, installed: 0.3.2]
      - filelock [required: >=3.0.0,<4, installed: 3.0.12]
      - six [required: >=1.9.0,<2, installed: 1.16.0]

I'm going to close this issue and reopen it again if issue will arise in the future. But at the moment it seems automagically resolved to me.

[RonnyPfannschmidt]

i'd like to leave this open until @nicoddemus leans in with a opinion on the library detail/workaround

im glad it resolved, it may be a dependency just dropping the broken pyreadline due to the windows3.10 issues

[nicoddemus]

The original PR for reference: #1281.

If this problem shows up again, I think we can add a guard for Windows 10 on _readline_workaround:

    if sys.platform.startswith("win32") and sys.version_info[:2] < (3, 10):
        try:
            import readline  # noqa: F401
        except ImportError:
            pass

But I think we can close this for now and wait and see.

[sergeyklay]

Thanks for the clarification. Let's wait and see