DEPR/Community: spin off s3 support

[jbrockmendel]

A disproportionate amount of our CI failures are in s3 tests e.g. test_with_s3_url. Moreover we have 2 xfailed tests with strict=False: test_write_s3_parquet_fails and test_write_s3_csv_fails xref #39155 where AFAICT we have no idea how to fix them.

This suggests we might not be the right team to maintain this feature. Is there a better home for this?

(The other big pain point CI-wise is the test_user_agent tests which cause all the timeouts)

[twoertwein]

Pandas uses fsspec to support s3 and all other kinds of protocols. Instead of testing s3 we could use any(?) other fsspec protocol that seems more reliable.

[jreback]

this sounds like a great idea

IIRC we have a lot of these tests prior to using fssc so certainly can use a non network protocol

[mroeschke]

I think the specific reason why those s3 tests fail is because the s3_resource fixture populates and deletes files from "s3" (moto) after each test, and in a muti-process CI environment this leads to flaky race condition errors. Running these tests in a single process environment would probably fix them: #44584

That being said, +1 to just using fsspec instead of s3f3 & gcsfs

[twoertwein]

If #44584 doesn't work out:

Migrating frequently failing s3 tests to other fsspec protocols sounds like a good idea to me. I think we shouldn't remove all s3/gcfs tests:

• There seems to be a bit of s3-specific code:

  pandas/pandas/io/common.py

  Line 360 in e08ffd4

  if filepath_or_buffer.startswith("s3a://"):

• And it might be good to keep at least four s3 tests: read_csv/to_csv both w/wo compression to make sure that text/binary mode is correctly inferred.

[jbrockmendel]

No momentum for spin-off, closing.