Non-deterministic test collection order for deterministic fixtures makes xdist fail

[aldanor]

Issues that may be related: #437, #594, #596, #669.

pytest-xdist seems to fail sometimes because it orders parametrized fixtures differently -- e.g., when a fixture depends on two or more parametrized fixtures. Using the latest pytest and pytest-xdist on Python 3.4 with this test script:

import pytest

@pytest.fixture(params=['foo', 'bar'])
def fixture1(request):
    return request.param

@pytest.fixture(params=['baz', 'foo'])
def fixture2(request):
    return request.param

def test_1(fixture1, fixture2):
    pass

Sometimes it works:

$ py.test -v -n 2
============================ test session starts ============================
platform linux -- Python 3.4.3 -- py-1.4.30 -- pytest-2.7.2 -- /home/test/env/bin/python3
plugins: xdist
[gw0] linux Python 3.4.3 cwd: /home/test/foo
[gw1] linux Python 3.4.3 cwd: /home/test/foo
[gw1] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
[gw0] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
gw0 [4] / gw1 [4]
scheduling tests via LoadScheduling

foo/test_1.py::test_1[2-3]
foo/test_1.py::test_1[1-3]
[gw0] PASSED foo/test_1.py::test_1[2-3]
[gw1] PASSED foo/test_1.py::test_1[1-3]
foo/test_1.py::test_1[1-4]
foo/test_1.py::test_1[2-4]
[gw0] PASSED foo/test_1.py::test_1[2-4]
[gw1] PASSED foo/test_1.py::test_1[1-4]

========================= 4 passed in 0.60 seconds ==========================

And sometimes it doesn't (because it chose to sort the fixtures by the second parameter first for some reason):

$ py.test -v -n 2
============================ test session starts ============================
platform linux -- Python 3.4.3 -- py-1.4.30 -- pytest-2.7.2 -- /home/test/env/bin/python3
plugins: xdist
[gw0] linux Python 3.4.3 cwd: /home/test/foo
[gw1] linux Python 3.4.3 cwd: /home/test/foo
[gw1] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
[gw0] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
gw0 [4] / gw1 [4]
scheduling tests via LoadScheduling
collecting 0 items / 1 errors
================================== ERRORS ===================================
___________________________ ERROR collecting gw0 ____________________________
Different tests were collected between gw1 and gw0. The difference is:
--- gw1

+++ gw0

@@ -1,4 +1,4 @@

 foo/test_1.py::test_1[foo-foo]
+foo/test_1.py::test_1[foo-bar]
+foo/test_1.py::test_1[bar-bar]
 foo/test_1.py::test_1[bar-foo]
-foo/test_1.py::test_1[bar-bar]
-foo/test_1.py::test_1[foo-bar]
========================== 1 error in 0.65 seconds ==========================

[RonnyPfannschmidt]

interesting find, my initial guess is, this is a artifact of the python hash-seed and the fixture management is just sorted different in some mappings

it'll take a while to figure the details, and it'll happen after the next release of xdist (which i am working on)

[aldanor]

Yea, the test collection mechanism is a bit flaky – in a much larger example which I won't be able to reproduce here I was able to work around it by changing some fixtures from session-scoped to function-scoped, I have no clue why that helped. Also, this doesn't seem to happen on Python 2 (not that I've seen it fail, at least).

By the way, I wonder why should it care about the order at all if the sets are the same?

[RonnyPfannschmidt]

That goes a few years back, I recall its a workaround for duplicate test ids or something like that

[jstasiak]

@aldanor FYI I fail to be able to reproduce the failure using the example code you provided. My software: Python 2.7.9 and 3.5.1, pytest 2.9.1, xdist-1.14.

[nicoddemus]

Hmm perhaps we can pass master's PYTHON_SEED value to workers? This way dict hashing should be the same across master and workers. This does not actually fix the fact that xdist uses indices to distribute tests, but at least seems sensible and would solve this issue (and others related).

[hectorv]

I'm seeing this while trying to move away from nosetests on a legacy test suite:

$ py.test -sxv -rw -n4 common/tests

this is my environment:

Python 2.7.11
pytest==2.9.2
pytest-xdist==1.14

Edit: to be clear, this works fine:

$ py.test -sxv -rw common/tests