BUG: pd.to_csv(..., compression="zip") creates multiple files in zip archive that cannot be read by pd.read_csv()

[tomstesco]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

import pandas as pd

# enough data to cause chunking into multiple files
n_data = 100000
df = pd.DataFrame(
    {'name': ["Raphael"]*n_data,
    'mask': ["red"]*n_data,
    'weapon': ["sai"]*n_data,
    }
)

compression_opts = dict(method='zip', archive_name='out.csv')
df.to_csv('out.csv.zip', index=False, compression=compression_opts)

# reading back the data produces an error
r_df = pd.read_csv("out.csv.zip")

# passing in compression_opts doesn't work either
r_df = pd.read_csv("out.csv.zip", compression=compression_opts)

How pandas was installed:

~/projects/testing$ ~/.pyenv/versions/3.8.6/bin/python3.8 -m venv text_venv
~/projects/testing$ . text_venv/bin/activate
(text_venv) ~/projects/testing$ pip install pandas==1.2.0

Problem description

Introduced in 1.2.0, to_csv with compression="zip" has an issue with chunking of data into multiple data files within the zip archive, when reading back the data this produces an error. When the data is very small (for example if n_data above was 2), this produces only one data file in the archive and no error when reading the archive.

This has been reported by other users on StackOverflow here: https://stackoverflow.com/questions/65689941/when-using-pandas-dataframe-to-csv-with-compression-zip-it-creates-a-zip-f

Expected Output

The to_csv should work with read_csv using zip compression. The behaviour with small data and medium sized data should be the same. The archive should always contain a single file or be read into a single df.

Output of pd.show_versions()

Python 3.8.6 (default, Nov 25 2020, 11:35:37)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import pandas as pd
pd.show_versions()
/Users/tom.s/projects/testing/text_venv/lib/python3.8/site-packages/setuptools/distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first.
warnings.warn(

INSTALLED VERSIONS

commit : 3e89b4c
python : 3.8.6.final.0
python-bits : 64
OS : Darwin
OS-release : 18.7.0
Version : Darwin Kernel Version 18.7.0: Tue Nov 10 00:07:31 PST 2020; root:xnu-4903.278.51~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.2.0
numpy : 1.19.5
pytz : 2020.5
dateutil : 2.8.1
pip : 20.2.1
setuptools : 49.2.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[twoertwein]

This should be fixed in 1.2.1 #38728. Do you have the same behavior on master?

[tomstesco]

Thanks for linking the PR, this does resolve the issue in current master.

Python 3.8.6 (default, Nov 25 2020, 11:35:37)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> pd.__version__
'1.3.0.dev0+449.g963cf2b5a'
>>> # enough data to cause chunking into multiple files
>>> n_data = 100000
>>> df = pd.DataFrame(
...     {'name': ["Raphael"]*n_data,
...     'mask': ["red"]*n_data,
...     'weapon': ["sai"]*n_data,
...     }
... )
>>>
>>> compression_opts = dict(method='zip', archive_name='out.csv')
>>> df.to_csv('out.csv.zip', index=False, compression=compression_opts)
>>>
>>> # reading back the data produces an error
>>> r_df = pd.read_csv("out.csv.zip")
>>> r_df
          name mask weapon
0      Raphael  red    sai
1      Raphael  red    sai
2      Raphael  red    sai
3      Raphael  red    sai
4      Raphael  red    sai
...        ...  ...    ...
99995  Raphael  red    sai
99996  Raphael  red    sai
99997  Raphael  red    sai
99998  Raphael  red    sai
99999  Raphael  red    sai

[100000 rows x 3 columns]

Output of pd.show_versions()

/Users/tom.s/projects/testing/test_venv/lib/python3.8/site-packages/setuptools/distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first. warnings.warn(

INSTALLED VERSIONS

commit : 963cf2b
python : 3.8.6.final.0
python-bits : 64
OS : Darwin
OS-release : 18.7.0
Version : Darwin Kernel Version 18.7.0: Tue Nov 10 00:07:31 PST 2020; root:xnu-4903.278.51~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.3.0.dev0+449.g963cf2b5a
numpy : 1.19.5
pytz : 2020.5
dateutil : 2.8.1
pip : 20.2.1
setuptools : 49.2.1
Cython : 0.29.21
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[simonjayhawkins]

this does resolve the issue in current master.

thanks for confirming

[simonjayhawkins]

duplicate of #38714