AWS_S3_HOST environment variable no longer available

[jakobkogler]

In the What's New - Other enhancements section from the 18.0 release it says that you can define the environment variable AWS_S3_HOST.

Therefore, when setting this environment variable, together with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY I expected the following code to work.

import pandas as pd
df = pd.DataFrame(dict(a=[1, 2, 3], b=[4, 5, 6]))
df.to_csv(fs.open('s3://bucket/key', mode='w'))

The script crashes with:

Exception ignored in: <bound method S3File.__del__ of <S3File bucket/key>>
Traceback (most recent call last):
  File "/home/xxx/.local/share/virtualenvs/xxx/lib/python3.6/site-packages/s3fs/core.py", line 1518, in __del__
    self.close()
  File "/home/xxx/.local/share/virtualenvs/xxx/lib/python3.6/site-packages/s3fs/core.py", line 1496, in close
    raise_from(IOError('Write failed: %s' % self.path), e)
  File "<string>", line 3, in raise_from
OSError: Write failed: bucket/key

This is the exact same error as when I don't specify the variable AWS_S3_HOST at all.

I managed to get a working solution by directly interacting directly with s3fs:

import os
import pandas as pd
import s3fs

fs = s3fs.S3FileSystem(client_kwargs={'endpoint_url': os.environ["AWS_S3_HOST"]})
df = pd.DataFrame(dict(a=[1, 2, 3], b=[4, 5, 6]))
df.to_csv(fs.open('s3://bucket/key', mode='w'))

Looks like this feature got (accidentally?) removed in https://github.com/pandas-dev/pandas/commit/dc4b0708f36b971f71890bfdf830d9a5dc019c7b#diff-a37b395bed03f0404dec864a4529c97dL94, when they switched from boto to s3fs.
Is it planned to support this environment variable again?
I'm sure that many companies, e.g. like the one I work for, hosts their own s3 server, e.g. with Minio, and don't use Amazon.

[WillAyd]

This is pretty similar to #16662 - I don't think this is something we'd generally offer support for so using the file system directly as you have in your solution is probably the way to go.

@TomAugspurger in case he has differing thoughts

[xieqihui]

+1 Support for this feature. It would be good to support customised S3 endpoint as it is widely used.

I have created a pull request to support this.

[MarcoGorelli]

Closing because, as I gather from the discussion in #29050, this would be out-of-scope for pandas (please ping me if I've misunderstood and I'll reopen)

[TomAugspurger]

You can do this now with storage_options. See how we do it in our tests:

pandas/pandas/tests/io/conftest.py

Lines 36 to 67 in 132e191

	@pytest.fixture
	def s3so(worker_id):
	worker_id = "5" if worker_id == "master" else worker_id.lstrip("gw")
	return dict(client_kwargs={"endpoint_url": f"http://127.0.0.1:555{worker_id}/"})
	@pytest.fixture(scope="session")
	def s3_base(worker_id):
	"""
	Fixture for mocking S3 interaction.
	Sets up moto server in separate process
	"""
	pytest.importorskip("s3fs")
	pytest.importorskip("boto3")
	requests = pytest.importorskip("requests")
	with tm.ensure_safe_environment_variables():
	# temporary workaround as moto fails for botocore >= 1.11 otherwise,
	# see https://github.com/spulec/moto/issues/1924 & 1952
	os.environ.setdefault("AWS_ACCESS_KEY_ID", "foobar_key")
	os.environ.setdefault("AWS_SECRET_ACCESS_KEY", "foobar_secret")
	pytest.importorskip("moto", minversion="1.3.14")
	pytest.importorskip("flask") # server mode needs flask too
	# Launching moto in server mode, i.e., as a separate process
	# with an S3 endpoint on localhost
	worker_id = "5" if worker_id == "master" else worker_id.lstrip("gw")
	endpoint_port = f"555{worker_id}"
	endpoint_uri = f"http://127.0.0.1:{endpoint_port}/"

If anyone wants to turn that into a doc example we'd welcome it.