Description
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
(optional) I have confirmed this bug exists on the master branch of pandas.
Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.
Code Sample, a copy-pastable example
import pandas as pd
# Read 900-line compressed dataset correctly
pd.read_sas('date_working.sas7bdat')
# Read 1000-line uncompressed dataset correctly
pd.read_sas('date_uncomp.sas7bdat')
# Read 1000-line compressed dataset - crashes (and leaves file open ...)
pd.read_sas('date_broken.sas7bdat')Problem description
I have a 1000-row SAS dataset (written by SAS 9.4 TS Level 1M6 on X64_SRV16 platform) which pandas chokes on reading. I attach the problematic dataset, as well as the same dataset uncompressed and a smaller subset of the compressed dataset that can be correctly read.
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\bug\venv\lib\site-packages\pandas\io\sas\sasreader.py", line 131, in read_sas
data = reader.read()
File "C:\Users\bug\venv\lib\site-packages\pandas\io\sas\sas7bdat.py", line 695, in read
rslt = self._chunk_to_dataframe()
File "C:\Users\bug\venv\lib\site-packages\pandas\io\sas\sas7bdat.py", line 739, in _chunk_to_dataframe
rslt[name] = self._byte_chunk[jb, :].view(dtype=self.byte_order + "d")
File "C:\Users\bug\venv\lib\site-packages\pandas\core\frame.py", line 3037, in setitem
self._set_item(key, value)
File "C:\Users\bug\venv\lib\site-packages\pandas\core\frame.py", line 3113, in _set_item
value = self._sanitize_column(key, value)
File "C:\Users\bug\venv\lib\site-packages\pandas\core\frame.py", line 3758, in _sanitize_column
value = sanitize_index(value, self.index)
File "C:\Users\bug\venv\lib\site-packages\pandas\core\internals\construction.py", line 748, in sanitize_index
"Length of values "
ValueError: Length of values (1000) does not match length of index (527)
Expected Output
No crash
Output of pd.show_versions()
INSTALLED VERSIONS
commit : d9fff27
python : 3.7.1.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.17763
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 9, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None
pandas : 1.1.0
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 10.0.1
setuptools : 39.0.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : 3.0.3
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.4.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : 1.2.0
xlwt : 1.3.0
numba : None