test_solve - hypothesis.errors.InvalidArgument: min_size=8_325 is larger than Hypothesis is designed to handle while generating 'x2'

[asmeurer]

__________________________________ test_solve __________________________________

    @pytest.mark.xp_extension('linalg')
>   @given(*solve_args())

array_api_tests/test_linalg.py:616: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <[AttributeError("'ListStrategy' object has no attribute 'element_strategy'") raised in repr()] ListStrategy object at 0x7eff05717450>
elements = builds(complex, FloatStrategy(min_value=-1.8446742974197924e+19, max_value=1.8446742974197924e+19, allow_nan=False, sm...9, max_value=1.8446742974197924e+19, allow_nan=False, smallest_nonzero_magnitude=1.1754943508222875e-38).map(downcast))
min_size = 8325, max_size = 8325

    def __init__(self, elements, min_size=0, max_size=float("inf")):
        super().__init__()
        self.min_size = min_size or 0
        self.max_size = max_size if max_size is not None else float("inf")
        assert 0 <= self.min_size <= self.max_size
        if min_size > BUFFER_SIZE:
>           raise InvalidArgument(
                f"min_size={min_size:_d} is larger than Hypothesis is designed to handle"
            )
E           hypothesis.errors.InvalidArgument: min_size=8_325 is larger than Hypothesis is designed to handle
E           while generating 'x2' from one_of(floating_dtypes(), complex_dtypes()).flatmap(lambda d: arrays(d, *args, elements=elements, **kwargs))
E           
E           You can reproduce this example by temporarily adding @reproduce_failure('6.98.13', b'AXicU2VlYGBkZEAHjCCoCwAFRABd') as a decorator on your test case

/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/hypothesis/strategies/_internal/collections.py:147: InvalidArgument

I'm not completely clear from the information given what the issue is. I don't see any issues with the arrays generated for test_solve https://github.com/data-apis/array-api-tests/blob/master/array_api_tests/test_linalg.py#L592, nor have I seen this error before.

[asmeurer]

Was able to reproduce in a debugger, but this is deep in hypothesis code. This looks like it's probably a hypothesis bug? Most likely one introduced since 6.97.1, which was the version I was previously using locally.

It seems to be having an issue with generating a list of 8000 elements in order to generate an array, but we generate arrays of that size all the time (our MAX_ARRAY_SIZE is 10000).

If this really is an issue with my strategy and not a hypothesis bug, there is a meta-hypothesis issue in that the error is completely incoherent.

[asmeurer]

Nothing stands out to me in the changelog, but I ran tens of thousands of examples in 6.97.1 and didn't get this error.

[asmeurer]

Also confirmed that @reproduce_failure('6.97.1', b'AXicU2VlYGBkZEAHjCCoCwAFRABd') does not reproduce the failure.

[asmeurer]

By the way, to reproduce this you may need to run against data-apis/array-api-compat#93 (numpy backend)

[asmeurer]

Seems like this was a recent change in hypothesis, but maybe the arrays() strategy was never updated HypothesisWorks/hypothesis#3896? I'm still confused why my solve strategy only is hitting this. CC @Zac-HD

[asmeurer]

Can you please share the code in question, preferably a minimal reproducing example?

So you see what I mean about the error message being unhelpful? :)

To reproduce, checkout this branch data-apis/array-api-compat#93, then run

PYTHONPATH=path/to/array-api-compat ARRAY_API_TESTS_MODULE=array_api_compat.numpy pytest array_api_tests/test_linalg.py -k solve

in this repo.

I linked to the relevant strategy above. I don't think there's any unique or anything. Just generating two arrays with related shapes.

[Zac-HD]

I've now spent more than an hour unsuccesfully trying to observe this error.

Can you give me a single-file reproducer with minimal dependencies?

[asmeurer]

I'll see what I can do on Monday. There are definitely a lot of moving parts here. It might be best to wait for @honno. I believe he is on vacation until late next week.

[asmeurer]

For me,

ARRAY_API_TESTS_MODULE=numpy.array_api pytest array_api_tests/test_linalg.py -k solve --max-examples=1000

reproduces the error. No need to use array-api-compat. This time the error looked like

_______________________________________________________________________________ test_solve ________________________________________________________________________________

    @pytest.mark.xp_extension('linalg')
>   @given(*solve_args())

array_api_tests/test_linalg.py:619:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <[AttributeError("'ListStrategy' object has no attribute 'element_strategy'") raised in repr()] ListStrategy object at 0x16a1fe710>
elements = builds(complex, FloatStrategy(min_value=-1.8446742974197924e+19, max_value=1.8446742974197924e+19, allow_nan=False, sm...9, max_value=1.8446742974197924e+19, allow_nan=False, smallest_nonzero_magnitude=1.1754943508222875e-38).map(downcast))
min_size = 8288, max_size = 8288

    def __init__(self, elements, min_size=0, max_size=float("inf")):
        super().__init__()
        self.min_size = min_size or 0
        self.max_size = max_size if max_size is not None else float("inf")
        assert 0 <= self.min_size <= self.max_size
        if min_size > BUFFER_SIZE:
>           raise InvalidArgument(
                f"min_size={min_size:_d} is larger than Hypothesis is designed to handle"
            )
E           hypothesis.errors.InvalidArgument: min_size=8_288 is larger than Hypothesis is designed to handle
E           while generating 'x2' from shared(sampled_from([(dtype('float32'), dtype('float32')), (dtype('float64'), dtype('float64')), (dtype('complex64'), dtype('complex64')), (dtype('complex128'), dtype('complex128')), (dtype('float32'), dtype('float64')), (dtype('float64'), dtype('float32')), (dtype('complex64'), dtype('complex128')), (dtype('complex128'), dtype('complex64'))])).map(lambda pair: pair[1]).flatmap(lambda d: arrays(d, *args, elements=elements, **kwargs))

../../miniconda3/envs/array-apis/lib/python3.11/site-packages/hypothesis/strategies/_internal/collections.py:147: InvalidArgument

This is with hypothesis 6.98.17

[Zac-HD]

If this really is an issue with my strategy and not a hypothesis bug, there is a meta-hypothesis issue in that the error is completely incoherent.

The reason I'm asking for a single-file repro is to have a small enough volume of code that I can tell why you're hitting this error. I'm fairly confident that it's inside your strategy, but knowing how will help improve the message etc.

...and while staring at that message again, I think part of the problem is that we're raising an exception before assigning all the attributes used in the __repr__, such that the exception message crashes out in a rather unhelpful way. I'm away from my dev setup this week, but if you want to move that if-clause from the middle to the end of ListStrategy.__init__ we should get a more informative message.