Description
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
(optional) I have confirmed this bug exists on the master branch of pandas.
Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.
Code Sample, a copy-pastable example
I have a few weird csv files with null bytes in the first row. Since 1.2.0 they are raising even if I skip the first row,
pd.read_csv("test.csv", engine="python", skiprows=[0])
pd.read_csv("test.csv", engine="c", skiprows=[0])Could not get this to fail with an example created in code.
Problem description
Both engines raising
Traceback (most recent call last):
File "/home/developer/.config/JetBrains/PyCharm2020.3/scratches/scratch_4.py", line 377, in <module>
pd.read_csv("/media/sf_Austausch/test.csv", engine="c", skiprows=[0], nrows=2)
File "/home/developer/PycharmProjects/pandas/pandas/io/parsers.py", line 605, in read_csv
return _read(filepath_or_buffer, kwds)
File "/home/developer/PycharmProjects/pandas/pandas/io/parsers.py", line 457, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "/home/developer/PycharmProjects/pandas/pandas/io/parsers.py", line 814, in __init__
self._engine = self._make_engine(self.engine)
File "/home/developer/PycharmProjects/pandas/pandas/io/parsers.py", line 1045, in _make_engine
return mapping[engine](self.f, **self.options) # type: ignore[call-arg]
File "/home/developer/PycharmProjects/pandas/pandas/io/parsers.py", line 1894, in __init__
self._reader = parsers.TextReader(self.handles.handle, **kwds)
File "pandas/_libs/parsers.pyx", line 517, in pandas._libs.parsers.TextReader.__cinit__
File "pandas/_libs/parsers.pyx", line 619, in pandas._libs.parsers.TextReader._get_header
File "pandas/_libs/parsers.pyx", line 813, in pandas._libs.parsers.TextReader._tokenize_rows
File "pandas/_libs/parsers.pyx", line 1942, in pandas._libs.parsers.raise_parser_error
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe4 in position 8: invalid continuation byte
Process finished with exit code 1
This worked on 1.1.5.
Relevant changes are:
https://github.com/pandas-dev/pandas/blob/9b16b1e1ee049b042a7d59cddd8fbd913137223f/pandas/io/common.py#L556:L557
infers encoding now, which leads to
https://github.com/pandas-dev/pandas/blob/9b16b1e1ee049b042a7d59cddd8fbd913137223f/pandas/io/common.py#L643:L651
where errors is always strict for read_csv. On 1.1.5 in case of no encoding given, errors was set to replace, which caused this to work. Was this an intended change? Labeling as Regression for now.
Expected Output
! xyz 0 1
0 * 2000 1 2
1 * 2001 0 0
cc @gfyoung @twoertwein
This was caused by #36997
Output of pd.show_versions()
INSTALLED VERSIONS
commit : cd0224d
python : 3.8.6.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-58-generic
Version : #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.3.0.dev0+356.ge0cb09e917
numpy : 1.19.2
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.3
setuptools : 49.6.0.post20201009
Cython : 0.29.21
pytest : 6.1.1
hypothesis : 5.37.1
sphinx : 3.2.1
blosc : None
feather : None
xlsxwriter : 1.3.7
lxml.etree : 4.5.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.18.1
pandas_datareader: None
bs4 : 4.9.3
bottleneck : 1.3.2
fsspec : 0.8.3
fastparquet : 0.4.1
gcsfs : 0.7.1
matplotlib : 3.3.2
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.5
pandas_gbq : None
pyarrow : 2.0.0
pyxlsb : None
s3fs : 0.4.2
scipy : 1.5.2
sqlalchemy : 1.3.20
tables : 3.6.1
tabulate : 0.8.7
xarray : 0.16.1
xlrd : 1.2.0
xlwt : 1.3.0
numba : 0.51.2