Skip to content

EHN: to_csv compression accepts file-like object #21249

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 30, 2018
Merged

EHN: to_csv compression accepts file-like object #21249

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
43c6c79
accept filehandle for to_csv compression
minggli May 29, 2018
00712b5
test filehandle compression for to_csv
minggli May 29, 2018
799596f
get filesize outside context manager
minggli May 29, 2018
6a90205
update docs
minggli May 29, 2018
ebf9d28
refactor test
minggli May 30, 2018
8d0c45b
add whatsnew
minggli May 30, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions pandas/core/frame.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1689,8 +1689,7 @@ def to_csv(self, path_or_buf=None, sep=",", na_rep='', float_format=None,
defaults to 'ascii' on Python 2 and 'utf-8' on Python 3.
compression : string, optional
A string representing the compression to use in the output file.
Allowed values are 'gzip', 'bz2', 'zip', 'xz'. This input is only
used when the first argument is a filename.
Allowed values are 'gzip', 'bz2', 'zip', 'xz'.
line_terminator : string, default ``'\n'``
The newline character or character sequence to use in the output
file
Expand Down
3 changes: 1 addition & 2 deletions pandas/core/series.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3761,8 +3761,7 @@ def to_csv(self, path=None, index=True, sep=",", na_rep='',
non-ascii, for python versions prior to 3
compression : string, optional
A string representing the compression to use in the output file.
Allowed values are 'gzip', 'bz2', 'zip', 'xz'. This input is only
used when the first argument is a filename.
Allowed values are 'gzip', 'bz2', 'zip', 'xz'.
date_format: string, default None
Format string for datetime objects.
decimal: string, default '.'
Expand Down
4 changes: 2 additions & 2 deletions pandas/io/formats/csvs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,9 +154,9 @@ def save(self):
# GH 17778 handles compression for byte strings.
if not close and self.compression:
f.close()
with open(self.path_or_buf, 'r') as f:
with open(f.name, 'r') as f:
data = f.read()
f, handles = _get_handle(self.path_or_buf, self.mode,
f, handles = _get_handle(f.name, self.mode,
encoding=encoding,
compression=self.compression)
f.write(data)
Expand Down
21 changes: 21 additions & 0 deletions pandas/tests/test_common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,3 +241,24 @@ def test_compression_size(obj, method, compression):
getattr(obj, method)(filename, compression=None)
uncompressed = os.path.getsize(filename)
assert uncompressed > compressed


@pytest.mark.parametrize('obj', [
DataFrame(100 * [[0.123456, 0.234567, 0.567567],
[12.32112, 123123.2, 321321.2]],
columns=['X', 'Y', 'Z']),
Series(100 * [0.123456, 0.234567, 0.567567], name='X')])
@pytest.mark.parametrize('method', ['to_csv'])
def test_compression_size_fh(obj, method, compression):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you make a new fixture alongside the existing to just have the compression options (exclude None), e.g. maybe compression_only (and can change the above test to use as well)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah sure.

if not compression:
pytest.skip("only test compression case.")

with tm.ensure_clean() as filename:
with open(filename, 'w') as fh:
getattr(obj, method)(fh, compression=compression)
compressed = os.path.getsize(filename)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you asssert that the fh is still open (inside the with block)?

Copy link
Contributor Author

@minggli minggli May 30, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 not sure how to do it here. with compression, the original handle is first written and closed and then compressed handle is created in the same name with entirety of strings of original handle enclosed. so assert fh.closed will pass inside the with block at the moment.

with tm.ensure_clean() as filename:
with open(filename, 'w') as fh:
getattr(obj, method)(fh, compression=None)
uncompressed = os.path.getsize(filename)
assert uncompressed > compressed