Read and write compressed CSVs to S3

[JasonSanchez]

pandas-dev/pandas#35129 was recently merged into pandas-dev:master.

fixes #22555: some compression algorithms did not set the mode to be writeable for file handles. Together with the re-factoring in the second commit, it is now possible to write to binary file handles with compression!

The comment below is unclear and makes it seem as if the feature were removed from pandas instead of that the feature will soon be available in pandas (and hopefully therefore in wrangler):

By now Pandas does not support in-memory CSV compression. pandas-dev/pandas#22555 So the compression will not be supported on Wrangler too.

Anyways, hoping this would be on your roadmap. Thanks for the great tool!

[igorborgest]

@JasonSanchez good to know about this PR, for sure it is something we have interest. We will add support for it as soon Pandas 1.2.0 became available.

P.S. If we have a new Wrangler's release before it, we will update/fix the comment mentioned.

Thanks!

[gvermillion]

As pandas 1.2.0 has been released, has there been any updates to when awswrangler will be able to write compressed CSVs directly to S3?

[igorborgest]

Hi @gvermillion @JasonSanchez

We've added support to it in the PR above 👆 .
Could you give it a try before the official release? You can install that directly from the dev branch:

pip install git+https://github.com/awslabs/aws-data-wrangler.git@write-compressed-text

[gvermillion]

Hey, @igorborgest. Thanks for the quick response!

I'll give this a go later this afternoon when I get pandas updated.