Different tests were collected between gw0 and gw1

[Alex-Chizhov]

I have a simple test with parametrization:

@pytest.mark.parametrize('product', testdata, ids=[repr(i) for i in testdata])
def test_add_new_product(appf_admin, product):
    with allure.step(f"Add product {product} in admin panel"):
        appf_admin.admin_panel.add_new_product(product)
    with allure.step(f"Get count of products {product} from search in admin panel"):
        name = re.match(r'name=(\S+)', str(product))
        clean_name = name.group(1)
        appf_admin.admin_panel.find_product_in_catalog_by_name(clean_name)
        products = appf_admin.admin_panel.get_count_product_row_from_catalog()
    with allure.step(f"Checking that searching product {product} return minimum 1 row with result"):
        assert products > 0

And simple generator of test data for parametrezation:

def random_string(maxlen):
    symbol = string.ascii_letters + string.digits
    return ''.join([random.choice(symbol) for i in range(random.randrange(1,maxlen))])

def random_digits(maxlen):
    symbol = string.digits
    return ''.join([random.choice(symbol) for i in range(random.randrange(1,maxlen))])

testdata_raw = [
Product(name=random_string(10), short_description=random_string(10), description=random_string(100),
        USD=random_digits(3)) for i in range(4)]
testdata = sorted(testdata_raw, key=lambda obj: obj.name)

When i launch test with additional argument:
-n 2
I get this error message:

============================= test session starts =============================
platform win32 -- Python 3.7.1, pytest-4.4.1, py-1.7.0, pluggy-0.9.0
rootdir: C: ...\Tests
plugins: xdist-1.28.0, parallel-0.0.9, forked-1.0.2, allure-pytest-2.6.0
gw0 I / gw1 I
gw0 [4] / gw1 [4]

gw1:None (gw1)
Different tests were collected between gw0 and gw1. The difference is:
--- gw0

+++ gw1

@@ -1,4 +1,4 @@

-test_add_new_product_with_param.py::test_add_new_product[name=LgysR6Y | USD=31 | id=None]
-test_add_new_product_with_param.py::test_add_new_product[name=hIx | USD=7 | id=None]
-test_add_new_product_with_param.py::test_add_new_product[name=lbpoI | USD=56 | id=None]
-test_add_new_product_with_param.py::test_add_new_product[name=pE | USD=51 | id=None]
+test_add_new_product_with_param.py::test_add_new_product[name=0fUz | USD=39 | id=None]
+test_add_new_product_with_param.py::test_add_new_product[name=b0heCg | USD=16 | id=None]
+test_add_new_product_with_param.py::test_add_new_product[name=sD | USD=8 | id=None]
+test_add_new_product_with_param.py::test_add_new_product[name=uHSt | USD=58 | id=None]

=================================== ERRORS ====================================
____________________________ ERROR collecting gw1 _____________________________
Different tests were collected between gw0 and gw1. The difference is:
--- gw0

+++ gw1

@@ -1,4 +1,4 @@

-test_add_new_product_with_param.py::test_add_new_product[name=LgysR6Y | USD=31 | id=None]
-test_add_new_product_with_param.py::test_add_new_product[name=hIx | USD=7 | id=None]
-test_add_new_product_with_param.py::test_add_new_product[name=lbpoI | USD=56 | id=None]
-test_add_new_product_with_param.py::test_add_new_product[name=pE | USD=51 | id=None]
+test_add_new_product_with_param.py::test_add_new_product[name=0fUz | USD=39 | id=None]
+test_add_new_product_with_param.py::test_add_new_product[name=b0heCg | USD=16 | id=None]
+test_add_new_product_with_param.py::test_add_new_product[name=sD | USD=8 | id=None]
+test_add_new_product_with_param.py::test_add_new_product[name=uHSt | USD=58 | id=None]
=========================== 1 error in 1.09 seconds ===========================
Process finished with exit code 0

Tell me please what could be the problem

[Alex-Chizhov]

As I understand it, xdist started test 2 times, and the test data generator also worked 2 times. Xdist doesn't like it. And we have to use a static JSON file with test data. We need to make the generation of this data in the fixture, not in the body of the test.

[Strilanc]

I am also running into this issue when using tests that do fuzzing. It would be very useful if there was a way to disable the "generating tests twice must produce identical tests" check. For example, given this test code in a file example.py:

@pytest.mark.parametrize('x,y', [
    (random.random(), random.random())
])
def test_example(x, y):
    assert isinstance(x, float)
    assert isinstance(y, float)

This command passes:

pytest example.py

And this command fails:

pytest example.py -n 2

With an error similar to this one:

Different tests were collected between gw0 and gw1. The difference is:
--- gw0

+++ gw1

@@ -1 +1 @@

-example.py::test_example[0.16380351559829032-0.38206603085139057]
+example.py::test_example[0.2613173472636646-0.7205939052389861]

Oddly, with one parameter instead of two the test will pass even with -n 2.

[nicoddemus]

Hi,

Sorry for the delay!

As I understand it, xdist started test 2 times, and the test data generator also worked 2 times. Xdist doesn't like it. And we have to use a static JSON file with test data. We need to make the generation of this data in the fixture, not in the body of the test.

Yes, each worker performs a standard collection, and sends the collected test ids (in order) back to the master node. The master node ensures every worker collected the same number of tests and in the same order, because the scheduler will from that point on send just the test indexes (not the entire node id) to each worker to tell them which test to execute. That's why the collection must be the same across all workers.

"generating tests twice must produce identical tests" check

As explained above, unfortunately it is not a simple check, but a central design point how the scheduler and workers interact.

[josiahls]

Hi, what is the solution to this? I also have:

@pytest.mark.parametrize("env", Envs.get_all_latest_envs())
def test_envs_all(env):
    gym.make(env)

where I am trying to test an agent on a bunch of environments. Different tests were collected between gw0 and gw1. Envs.get_all_latest_envs() just returns a list of env names. This works in the normal pytest execution, however some environments take forever to init, so I'd like to run others in parallel instead of waiting.

I'm getting a similar error.

plugins: xdist-1.29.0, forked-1.0.2, asyncio-0.10.0
gw0 I / gw1 I
gw0 [483] / gw1 [483]

gw1:None (gw1)
The difference is:
--- gw0

+++ gw1

@@ -1,483 +1,483 @@

+fast_rl/tests/test_Envs.py::test_envs_all[AmidarNoFrameskip-v40]
+fast_rl/tests/test_Envs.py::test_envs_all[AirRaidNoFrameskip-v40]
+fast_rl/tests/test_Envs.py::test_envs_all[maze-sample-5x5-v0]
+fast_rl/tests/test_Envs.py::test_envs_all[VideoPinballNoFrameskip-v40]
+fast_rl/tests/test_Envs.py::test_envs_all[AsteroidsNoFrameskip-v40]

[aganders3]

@josiahls Try using sorted(Env.get_all_latest_envs()). I had the same problem and the issue was each worker generating the "same" list but in a different order.

[therefromhere]

edit, moved my comment to a new issue #472

[federicosacerdoti]

Had this issue, fixed with sorting the parameters

@pytest.mark.parametrize("foo", sorted(foos))