Skip to content

df.to_dict('records') broken when 255+ columns present #24939

Closed
@ebolyen

Description

@ebolyen

Code Sample, a copy-pastable example if possible

Python 3.6.7 | packaged by conda-forge | (default, Nov 21 2018, 03:09:43) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.2.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pandas as pd                                                                                                                                                                                                 

In [2]: df = pd.DataFrame({('A_%d' % i): [i] for i in range(254)})                                                                                                                                                          

In [3]: r = df.to_dict('records')                                                                                                                                                                                           

In [4]: len(r)                                                                                                                                                                                                              
Out[4]: 1

In [5]: len(r[0])                                                                                                                                                                                                           
Out[5]: 254

In [6]: df_bad = pd.DataFrame({('A_%d' % i): [i] for i in range(255)})                                                                                                                                                      

In [7]: r_fail = df_bad.to_dict('records')                                                                                                                                                                                  
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-7-4dd3de569735> in <module>
----> 1 r_fail = df_bad.to_dict('records')

~/.conda/envs/pandas-0.24/lib/python3.6/site-packages/pandas/core/frame.py in to_dict(self, orient, into)
   1300                 into_c((k, com.maybe_box_datetimelike(v))
   1301                        for k, v in compat.iteritems(row._asdict()))
-> 1302                 for row in self.itertuples(index=False)]
   1303         elif orient.lower().startswith('i'):
   1304             if not self.index.is_unique:

~/.conda/envs/pandas-0.24/lib/python3.6/site-packages/pandas/core/frame.py in <listcomp>(.0)
   1300                 into_c((k, com.maybe_box_datetimelike(v))
   1301                        for k, v in compat.iteritems(row._asdict()))
-> 1302                 for row in self.itertuples(index=False)]
   1303         elif orient.lower().startswith('i'):
   1304             if not self.index.is_unique:

AttributeError: 'tuple' object has no attribute '_asdict'

Problem description

Our test suite is failing with the latest release (0.24.0) on conda-forge. After looking into it, I've discovered that when 255 or more columns exist, the to_dict('records') method fails with AttributeError: 'tuple' object has no attribute '_asdict'.

Conda info

(pandas-0.24) 15:04:03 ~
§ conda list
# packages in environment at /home/evan/.conda/envs/pandas-0.24:
#
# Name                    Version                   Build  Channel
backcall                  0.1.0                      py_0    conda-forge
blas                      1.0                         mkl  
ca-certificates           2018.11.29           ha4d7672_0    conda-forge
certifi                   2018.11.29            py36_1000    conda-forge
decorator                 4.3.0                      py_0    conda-forge
intel-openmp              2019.1                      144  
ipython                   7.2.0           py36h24bf2e0_1000    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
jedi                      0.13.2                py36_1000    conda-forge
libffi                    3.2.1             hf484d3e_1005    conda-forge
libgcc-ng                 7.3.0                hdf63c60_0    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3    conda-forge
libstdcxx-ng              7.3.0                hdf63c60_0    conda-forge
mkl                       2019.1                      144  
mkl_fft                   1.0.10           py36h14c3975_1    conda-forge
mkl_random                1.0.2            py36h637b7d7_2    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
numpy                     1.15.4           py36h7e9f1db_0  
numpy-base                1.15.4           py36hde5b4d6_0  
openssl                   1.0.2p            h14c3975_1002    conda-forge
pandas                    0.24.0           py36hf484d3e_0    conda-forge
parso                     0.3.1                      py_0    conda-forge
pexpect                   4.6.0                 py36_1000    conda-forge
pickleshare               0.7.5                 py36_1000    conda-forge
pip                       18.1                  py36_1000    conda-forge
prompt_toolkit            2.0.7                      py_0    conda-forge
ptyprocess                0.6.0                 py36_1000    conda-forge
pygments                  2.3.1                      py_0    conda-forge
python                    3.6.7             hd21baee_1001    conda-forge
python-dateutil           2.7.5                      py_0    conda-forge
pytz                      2018.9                     py_0    conda-forge
readline                  7.0               hf8c457e_1001    conda-forge
setuptools                40.6.3                   py36_0    conda-forge
six                       1.12.0                py36_1000    conda-forge
sqlite                    3.26.0            h67949de_1000    conda-forge
tk                        8.6.9             h84994c4_1000    conda-forge
traitlets                 4.3.2                 py36_1000    conda-forge
wcwidth                   0.1.7                      py_1    conda-forge
wheel                     0.32.3                   py36_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
zlib                      1.2.11            h14c3975_1004    conda-forge
(pandas-0.24) 15:04:04 ~
§ conda info

     active environment : pandas-0.24
    active env location : /home/evan/.conda/envs/pandas-0.24
            shell level : 1
       user config file : /home/evan/.condarc
 populated config files : 
          conda version : 4.5.12
    conda-build version : 3.17.6
         python version : 3.7.2.final.0
       base environment : /opt/Miniconda3  (read only)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/linux-64
                          https://repo.anaconda.com/pkgs/pro/noarch
          package cache : /opt/Miniconda3/pkgs
                          /home/evan/.conda/pkgs
       envs directories : /home/evan/.conda/envs
                          /opt/Miniconda3/envs
               platform : linux-64
             user-agent : conda/4.5.12 requests/2.19.1 CPython/3.7.2 Linux/4.20.3-arch1-1-ARCH antergos/ glibc/2.28
                UID:GID : 1000:985
             netrc file : None
           offline mode : False

Output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.7.final.0
python-bits: 64
OS: Linux
OS-release: 4.20.3-arch1-1-ARCH
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.0
pytest: None
pip: 18.1
setuptools: 40.6.3
Cython: None
numpy: 1.15.4
scipy: None
pyarrow: None
xarray: None
IPython: 7.2.0
sphinx: None
patsy: None
dateutil: 2.7.5
pytz: 2018.9
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

Metadata

Metadata

Assignees

No one assigned

    Labels

    RegressionFunctionality that used to work in a prior pandas version

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions