Skip to content

BUG: MultiIndex levels are not preserved for empty DataFrames after to_parquet roundtrip #59385

Open
@shunr

Description

@shunr

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
data = pd.DataFrame(index=pd.MultiIndex(levels=[['abc'], []], codes=[[], []]))
data.to_parquet('test.parquet')
data2 = pd.read_parquet('test.parquet')
print(data.index.levels)
> [['abc'], []]
print(data2.index.levels)
> [[], []]

Issue Description

When I create an empty DataFrame with an MultiIndex that has one or more empty levels, the non-empty levels are not preserved after saving the DataFrame to parquet and re-loading it.

Expected Behavior

I expect the index levels to stay the same after reading from parquet

import pandas as pd
data = pd.DataFrame(index=pd.MultiIndex(levels=[['abc'], []], codes=[[], []]))
data.to_parquet('test.parquet')
data2 = pd.read_parquet('test.parquet')
print(data.index.levels)
> [['abc'], []]
print(data2.index.levels)
> [['abc'], []]

Installed Versions

INSTALLED VERSIONS
------------------
commit           : 2e218d10984e9919f0296931d92ea851c6a6faf5
python           : 3.11.9.final.0
python-bits      : 64
OS               : Linux
OS-release       : 5.15.0-43-generic
Version          : #46~20.04.1-Ubuntu SMP Thu Jul 14 15:20:17 UTC 2022
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.utf8
LOCALE           : en_US.UTF-8

pandas           : 1.5.3
numpy            : 1.23.5
pytz             : 2024.1
dateutil         : 2.9.0
setuptools       : 69.5.1
pip              : 24.0
Cython           : None
pytest           : 8.3.1
hypothesis       : None
sphinx           : 7.2.6
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 5.2.2
html5lib         : None
pymysql          : None
psycopg2         : 2.9.9
jinja2           : 3.1.4
IPython          : 8.26.0
pandas_datareader: None
bs4              : 4.12.3
bottleneck       : None
brotli           : 1.1.0
fastparquet      : None
fsspec           : 2024.5.0
gcsfs            : 2024.5.0
matplotlib       : 3.9.1
numba            : 0.58.1
numexpr          : 2.10.0
odfpy            : None
openpyxl         : 3.1.4
pandas_gbq       : None
pyarrow          : 15.0.2
pyreadstat       : None
pyxlsb           : None
s3fs             : 2024.5.0
scipy            : 1.14.0
snappy           : None
sqlalchemy       : 1.4.49
tables           : 3.9.2
tabulate         : 0.9.0
xarray           : 2024.3.0
xlrd             : 2.0.1
xlwt             : None
zstandard        : 0.23.0
tzdata           : 2024.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions