test_ftpfs fails on macOS with python 3.8.5 but works with python 3.7

[tricktron]

How to reproduce

1. Install python 3.8.5

$ curl -O https://files.pythonhosted.org/packages/1d/a1/8813629b38a8d97e8f1eceb6c7da03b37633c93104fbd8e30e09d195425a/fs-2.4.11.tar.gz
$ tar xf fs-2.4.11.tar.gz
$ cd  xf fs-2.4.11
$ virtualenv env
$ source env/bin/activate
$ python -m pip install -e .
$ pip install pytest pyftpdlib psutil mock
$ python -m pytest . # fails

Error log:

ERROR at teardown of TestFTPFS.test_writetext ___________________________________________________

cls = <class 'tests.test_ftpfs.TestFTPFS'>

    @classmethod
    def tearDownClass(cls):
>       cls.server.stop()

tests/test_ftpfs.py:166: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
env/lib/python3.8/site-packages/pyftpdlib/test/__init__.py:355: in stop
    assert_free_resources()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def assert_free_resources():
        ts = threading.enumerate()
        assert len(ts) == 1, ts
        p = psutil.Process()
        children = p.children()
        if children:
            for p in children:
                p.kill()
                p.wait(1)
>           assert not children, children
E           AssertionError: [psutil.Process(pid=8168, status='terminated', exitcode=<Negsignal.SIGKILL: -9>, started='07:30:07')]

env/lib/python3.8/site-packages/pyftpdlib/test/__init__.py:261: AssertionError

Python 3.8.5 error log:

/nix/store/cp9nlaz4a24n8sp9f738ngdnrd19s4rp-python3-3.8.5/lib/python3.8/multiprocessing/resource_tracker.py:96: UserWarning: resource_tracker: process died unexpectedly, relaunching.  Some resources might leak.
  warnings.warn('resource_tracker: process died unexpectedly, '
Traceback (most recent call last):
  File "/nix/store/cp9nlaz4a24n8sp9f738ngdnrd19s4rp-python3-3.8.5/lib/python3.8/multiprocessing/resource_tracker.py", line 201, in main
    cache[rtype].remove(name)
KeyError: '/mp-sr7swli_'
Traceback (most recent call last):
  File "/nix/store/cp9nlaz4a24n8sp9f738ngdnrd19s4rp-python3-3.8.5/lib/python3.8/multiprocessing/resource_tracker.py", line 201, in main
    cache[rtype].remove(name)
KeyError: '/mp-zuko65p_'
Traceback (most recent call last):
  File "/nix/store/cp9nlaz4a24n8sp9f738ngdnrd19s4rp-python3-3.8.5/lib/python3.8/multiprocessing/resource_tracker.py", line 201, in main
    cache[rtype].remove(name)
KeyError: '/mp-_i63v3_i'
Traceback (most recent call last):
  File "/nix/store/cp9nlaz4a24n8sp9f738ngdnrd19s4rp-python3-3.8.5/lib/python3.8/multiprocessing/resource_tracker.py", line 201, in main
    cache[rtype].remove(name)
KeyError: '/mp-6yd4e3mj'
Traceback (most recent call last):
  File "/nix/store/cp9nlaz4a24n8sp9f738ngdnrd19s4rp-python3-3.8.5/lib/python3.8/multiprocessing/resource_tracker.py", line 201, in main
    cache[rtype].remove(name)
KeyError: '/mp-xbyjd_kd'
Traceback (most recent call last):
  File "/nix/store/cp9nlaz4a24n8sp9f738ngdnrd19s4rp-python3-3.8.5/lib/python3.8/multiprocessing/resource_tracker.py", line 201, in main
    cache[rtype].remove(name)
KeyError: '/mp-u1yjocb0'

The test passes using python 3.7

System Info

• system: "x86_64-darwin"
• host os: Darwin 19.6.0, macOS 10.15.6

[althonos]

I can't reproduce on my local machine since it is running Linux. It may be a good idea to setup Travis-CI to test on OSX / Python 3.8 seeing the recrudescence of OSX/3.8 bug reports.

[tricktron]

@althonos That would be nice

[lurch]

@tricktron Could you also test with the latest master version of pyfilesystem downloaded from this repo, to see if that behaves any differently to the 2.4.11 version on PyPI?

Although looking at your error log it seems the error is actually coming from https://github.com/giampaolo/pyftpdlib rather than pyfilesystem? Pinging @giampaolo in case he has any ideas / suggestions.

[giampaolo]

Hey Andrew! Yep, looks like it's pyftpdlib (test) fault. Pyftpdlib tests are unstable (due to threads). I'll have to fix them sooner or later (most likely than sooner =)).

[tricktron]

@lurch Just for the sake of completeness: I also tested it with themaster version and the errors are identical.
@giampaolo Thanks for the clarification.

[marchostau]

This error is solved? I'm getting the same error using Linux and python3.9, 3.10.