Skip to content

CI: Unreliable Tests write to S3 #39155

Closed
@alimcmaster1

Description

@alimcmaster1

Seen this in a few PRs:

https://github.com/pandas-dev/pandas/pull/39153/checks?check_run_id=1697646268

________________________ TestS3.test_write_s3_csv_fails ________________________
[gw1] linux -- Python 3.7.9 /usr/share/miniconda/envs/pandas-dev/bin/python

self = <pandas.tests.io.parser.test_network.TestS3 object at 0x7feea42e2610>
tips_df =      total_bill   tip     sex smoker   day    time  size
0         16.99  1.01  Female     No   Sun  Dinner     2
1   ....75    Male     No   Sat  Dinner     2
243       18.78  3.00  Female     No  Thur  Dinner     2

[244 rows x 7 columns]
s3so = {'client_kwargs': {'endpoint_url': 'http://127.0.0.1:5551/'}}

    def test_write_s3_csv_fails(self, tips_df, s3so):
        # GH 32486
        # Attempting to write to an invalid S3 path should raise
        import botocore
    
        # GH 34087
        # https://boto3.amazonaws.com/v1/documentation/api/latest/guide/error-handling.html
        # Catch a ClientError since AWS Service Errors are defined dynamically
        error = (FileNotFoundError, botocore.exceptions.ClientError)
    
        with pytest.raises(error, match="The specified bucket does not exist"):
            tips_df.to_csv(
>               "s3://an_s3_bucket_data_doesnt_exit/not_real.csv", storage_options=s3so
            )
E           Failed: DID NOT RAISE (<class 'FileNotFoundError'>, <class 'botocore.exceptions.ClientError'>)

pandas/tests/io/parser/test_network.py:224: Failed
______________________ TestS3.test_write_s3_parquet_fails ______________________
[gw1] linux -- Python 3.7.9 /usr/share/miniconda/envs/pandas-dev/bin/python

self = <pandas.tests.io.parser.test_network.TestS3 object at 0x7feea435c790>
tips_df =      total_bill   tip     sex smoker   day    time  size
0         16.99  1.01  Female     No   Sun  Dinner     2
1   ....75    Male     No   Sat  Dinner     2
243       18.78  3.00  Female     No  Thur  Dinner     2

[244 rows x 7 columns]
s3so = {'client_kwargs': {'endpoint_url': 'http://127.0.0.1:5551/'}}

    @td.skip_if_no("pyarrow")
    def test_write_s3_parquet_fails(self, tips_df, s3so):
        # GH 27679
        # Attempting to write to an invalid S3 path should raise
        import botocore
    
        # GH 34087
        # https://boto3.amazonaws.com/v1/documentation/api/latest/guide/error-handling.html
        # Catch a ClientError since AWS Service Errors are defined dynamically
        error = (FileNotFoundError, botocore.exceptions.ClientError)
    
        with pytest.raises(error, match="The specified bucket does not exist"):
            tips_df.to_parquet(
                "s3://an_s3_bucket_data_doesnt_exit/not_real.parquet",
>               storage_options=s3so,
            )
E           Failed: DID NOT RAISE (<class 'FileNotFoundError'>, <class 'botocore.exceptions.ClientError'>)

Metadata

Metadata

Assignees

No one assigned

    Labels

    DependenciesRequired and optional dependenciesIO NetworkLocal or Cloud (AWS, GCS, etc.) IO IssuesTestingpandas testing functions or related to the test suite

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions