TST: Compression Inference Tests for read_*

[gfyoung]

xref: #17206 (comment)

cc @dhimmel

[dhimmel]

See #17206 (comment). In short, not all read_* methods use functionality from io.common. This would be nice, see #15008, but it's a huge task. If we tested compression inference for all read_* methods, many would likely fail... there's lot's of undesirable duplicated functionality across the read code.

[gfyoung]

Fair enough. Perhaps you can testing whichever functions hit the io.common path and revise your whatsnew entry to reflect that.

[jreback]

If we tested compression inference for all read_* methods, many would likely fail

@dhimmel but that's exactly the point. I am quite happy to have a comprehensive test, that xfails things that are not converted / implemented.

[dhimmel]

Looking through the source code, I believe io.common._infer_compression is only called in the following three places:

io.pickle.to_pickle()

pandas/pandas/io/pickle.py

Line 38 in a46e5be

inferred_compression = _infer_compression(path, compression)

io.pickle.read_pickle()

pandas/pandas/io/pickle.py

Line 76 in a46e5be

inferred_compression = _infer_compression(path, compression)

io.parsers._read()

pandas/pandas/io/parsers.py

Line 395 in a46e5be

compression = _infer_compression(filepath_or_buffer, compression)

The first two are for pickle IO. Tracking down where/how io.parsers._read gets used has been a bit more challenging. Will keep looking into it.

[dhimmel]

io.parsers._read is called by io.parsers._make_parser_function:

pandas/pandas/io/parsers.py

Line 660 in a46e5be

return _read(filepath_or_buffer, kwds)

_make_parser_function is then used in io.parsers to create read_csv and read_table

pandas/pandas/io/parsers.py

Lines 667 to 671 in a46e5be

	read_csv = _make_parser_function('read_csv', sep=',')
	read_csv = Appender(_read_csv_doc)(read_csv)
	read_table = _make_parser_function('read_table', sep='\t')
	read_table = Appender(_read_table_doc)(read_table)

Therefore, I believe, at the current time, _infer_compression is only used by the user-facing IO functions of read_pickle, write_pickle, read_csv, and read_table?

@gfyoung @jreback: should I revise the What's New Entry for #17206 to list these four functions?

pandas/doc/source/whatsnew/v0.21.0.txt

Line 129 in 6fe6832

- `read_*` methods can now infer compression from non-string paths, such as ``pathlib.Path`` objects (:issue:`17206`).

[gfyoung]

@dhimmel : Agreed, part of your PR should be to revise the whatsnew and add tests for any affected functions (and maybe even those that aren't affected but should eventually gain support).

[jreback]

@dhimmel any interest in working on this one would be great.