Compression keyword for Stata and others?

[ozak]

Hi,

I was trying to open zipped Stata files and thought one could do it as in read_csvusing the compression='zip' keyword option. Is this not implemented? Saw #15644 and follow ups, but at the time it seems the discussion was about output not input files. In many cases providers give Stata files in zip format, which e.g. in my case may mean 1000's of different zipped archives containing Stata + other files. #12103 seems to have added the functionality more generally, but at least in pd.__version__=0.23.4 it still complains when passing the compression keyword. I do not see the option in pandas/io/stata.py although in common.py the option seems to be there. Any pointers? I am happy to contribute if I can figure out where and how this is implemented for other formats.

[WillAyd]

Related issue #21640

[bashtage]

Compression on io function is for native compression. csv can be gziped so this looks like a native (and common) compression. Stata dta files do not support compression. All you need to do is to use zipfile.ZipFile to traverse the .dta files, and then you can read them into DataFrames using read_stata. This doesn't require (or justify) adding an option to read_stata that is supporting something that is orthogonal to the dta file format.

[ozak]

Since all kinds of files can be zipped and are distributed like that, it seems strange to implement it in one kind and not others. In any case, you are right that any zip file can be unzipped and loaded into a pandas data frame using zipfile.

[MaxGhenis]

This distinction also seems strange to me; Stata files are commonly zipped (e.g. the Federal Reserve's Survey of Consumer Finances, where zipping reduces filesize by 90%), and since it's possible, it seems more convenient to offer the same interface provided to read_csv (even if the underlying mechanism will differ).

I asked how to do this and got a couple answers using zipfile.ZipFile: https://stackoverflow.com/questions/59122596/read-a-zipped-stata-file-from-url-into-pandas

Adding the functionality to read_stata would save six lines of code when pulling the zip file from a URL.

[jzwinck]

I agree, it's not nice that read_csv and read_json and read_pickle all understand compression but read_stata does not. Zipped Stata files are popular, and there's a zipsave package to create and load them in their native habitat: https://ideas.repec.org/c/boc/bocode/s446702.html

Pandas should support this. @WillAyd would you consider re-opening this?