Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
# example 1
ser1 = pd.Series(pd.Categorical([None] * 5))
converted1 = ser1.convert_dtypes(dtype_backend="pyarrow")
# example 2
ser2 = pd.Series(pd.Categorical([None] * 5, categories=["S1", "S2"]))
converted2 = ser2.convert_dtypes(dtype_backend="pyarrow")Issue Description
Trying to convert categorical series with empty categories results in pyarrow.lib.ArrowNotImplementedError: NumPyConverter doesn't implement <null> conversion. .
Trying to convert categorical series with non-empty categories returns null[pyarrow] dtype.
This is inconsistent with regular behavior when converting categoricals - if series is not empty, convert_dtypes call returns categorical dtype (essentially ignoring the requested pyarrow backend conversion).
I encountered this issue when trying to read files (like parquet, spss, ...) where some categorical variables are empty...
Expected Behavior
Returning categorical dtype with propagated categories, and not raising errors.
Installed Versions
INSTALLED VERSIONS
commit : 00855f8
python : 3.12.0
python-bits : 64
OS : Windows
OS-release : 11
Version : 10.0.22631
machine : AMD64
processor : Intel64 Family 6 Model 165 Stepping 2, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252
pandas : 3.0.0.dev0+1536.g00855f81bd
numpy : 1.26.4
dateutil : 2.9.0.post0
pip : 24.2
Cython : 3.0.11
sphinx : 8.0.2
IPython : 8.27.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
blosc : None
bottleneck : 1.4.0
fastparquet : 2024.5.0
fsspec : 2024.9.0
html5lib : 1.1
hypothesis : 6.112.2
gcsfs : 2024.9.0post1
jinja2 : 3.1.4
lxml.etree : 5.3.0
matplotlib : 3.9.2
numba : 0.60.0
numexpr : 2.10.1
odfpy : None
openpyxl : 3.1.5
psycopg2 : 2.9.9
pymysql : 1.4.6
pyarrow : 17.0.0
pyreadstat : 1.2.7
pytest : 8.3.3
python-calamine : None
pytz : 2024.2
pyxlsb : 1.0.10
s3fs : 2024.9.0
scipy : 1.14.1
sqlalchemy : 2.0.35
tables : 3.10.1
tabulate : 0.9.0
xarray : 2024.9.0
xlrd : 2.0.1
xlsxwriter : 3.2.0
zstandard : 0.23.0
tzdata : 2024.2
qtpy : None
pyqt5 : None