Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
df = pd.DataFrame({'A': ['foo', 'bar', None], 'B': [1, 2, 3]})
grouped = df.groupby('A', dropna=False)
len(grouped) # raises exception with pandas 2.2.2; returns 3 with current main branchIssue Description
The above example raises the following exception with pandas 2.2.2:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/tmp/myen/lib/python3.11/site-packages/pandas/core/groupby/groupby.py", line 787, in __len__
return len(self.groups)
^^^^^^^^^^^
File "/tmp/myen/lib/python3.11/site-packages/pandas/core/groupby/groupby.py", line 851, in groups
return self._grouper.groups
^^^^^^^^^^^^^^^^^^^^
File "properties.pyx", line 36, in pandas._libs.properties.CachedProperty.__get__
File "/tmp/myen/lib/python3.11/site-packages/pandas/core/groupby/ops.py", line 717, in groups
return self.groupings[0].groups
^^^^^^^^^^^^^^^^^^^^^^^^
File "properties.pyx", line 36, in pandas._libs.properties.CachedProperty.__get__
File "/tmp/myen/lib/python3.11/site-packages/pandas/core/groupby/grouper.py", line 842, in groups
cats = Categorical.from_codes(self.codes, self._group_index, validate=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/myen/lib/python3.11/site-packages/pandas/core/arrays/categorical.py", line 745, in from_codes
dtype = CategoricalDtype._from_values_or_dtype(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/myen/lib/python3.11/site-packages/pandas/core/dtypes/dtypes.py", line 338, in _from_values_or_dtype
dtype = CategoricalDtype(categories, ordered)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/myen/lib/python3.11/site-packages/pandas/core/dtypes/dtypes.py", line 221, in __init__
self._finalize(categories, ordered, fastpath=False)
File "/tmp/myen/lib/python3.11/site-packages/pandas/core/dtypes/dtypes.py", line 378, in _finalize
categories = self.validate_categories(categories, fastpath=fastpath)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/myen/lib/python3.11/site-packages/pandas/core/dtypes/dtypes.py", line 576, in validate_categories
raise ValueError("Categorical categories cannot be null")
ValueError: Categorical categories cannot be null
My understanding of the API documentation is that len(grouped) should be 3. Indeed, this is the behavior of the current main branch. The change in behavior is due to #57595, which was supposed to only improve performance, but coincidentally fixed this bug.
As the bug is still present in the latest version (2.2.2), it may be worth backporting the (unintended) fix of #57595.
Expected Behavior
len(grouped) should be 3, as there are three groups: 'foo', 'bar', and None.
Installed Versions
pandas : 2.2.2
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0.post0
setuptools : 68.1.2
pip : 23.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None