Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
import pyarrow as pa
a = pa.array([1, 2, 3])
b = pa.array([[True, False, True]] * 3, type=pa.list_(pa.bool_(), 3))
table = pa.Table.from_arrays([a, b], names=['a', 'b'])
pa_table = table.to_pandas(types_mapper=pd.ArrowDtype)
pa_table.to_parquet("using_types_mapper.parquet")
data_frame = pd.read_parquet("using_types_mapper.parquet")
Issue Description
In the example, we convert the PyArrow table to a pandas DataFrame, using the types mapper to enforce the fixed_size_list
type of column b
.
As of the versions indicated below, this results in the following error:
NotImplementedError: Passing pyarrow type specific parameters ([3]) in the string is not supported. Please construct an ArrowDtype object with a pyarrow_dtype instance with specific parameters.
If you inspect the generated file, you see that the pyarrow schema reflects the fixed_size_list
. e.g pq.read_metadata("using_types_mapper.parquet").schema.to_arrow_schema()
.
If, instead, you do not indicate a types mapper, the type of fixed_size_list
is lost, and the parquet file on-disk just uses a list
type:
import pandas as pd
import pyarrow as pa
a = pa.array([1, 2, 3])
b = pa.array([[True, False, True]] * 3, type=pa.list_(pa.bool_(), 3))
table = pa.Table.from_arrays([a, b], names=['a', 'b'])
pa_table = table.to_pandas() # this is the different line
pa_table.to_parquet("loses_fixed_size.parquet")
data_frame = pd.read_parquet("loses_fixed_size.parquet")
display(pq.read_metadata("loses_fixed_size.parquet").schema.to_arrow_schema())
Expected Behavior
I would expect to
- read the parquet file without error
- for the returned dataframe to have some kind of list type for column
b
- for the parquet file's on-disk metadata to still include a
fixed_size_list
type for columnb
Installed Versions
INSTALLED VERSIONS
commit : d9cdd2e
python : 3.11.9.final.0
python-bits : 64
OS : Linux
OS-release : 6.5.0-1025-oem
Version : #26-Ubuntu SMP PREEMPT_DYNAMIC Tue Jun 18 12:35:22 UTC 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 2.2.2
numpy : 2.1.1
pytz : 2024.1
dateutil : 2.9.0
setuptools : 72.1.0
pip : 24.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 8.27.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 17.0.0
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None