Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
df = pd.DataFrame(
{
"col1": [None, 1, 1, 0, None, 0],
"col2": [4, 5, None, 7, 4, 5],
"col3": [0, 1, 2, 3, 4, 5]
}
)
# order of this is [(nan, 4.0), (0.0, 5.0), (0.0, 7.0), (1.0, 5.0), (1.0, nan)]
print(list(df.groupby(['col1', 'col2'], dropna=False).groups.keys()))
# But this is the correct order, with nulls always last:
# [(0.0, 5.0), (0.0, 7.0), (1.0, 5.0), (1.0, nan), (nan, 4.0)]
print(list(df.groupby(['col1', 'col2'], dropna=False).sum().index))
# similarly with sort=False, order is same:
# [(nan, 4.0), (0.0, 5.0), (0.0, 7.0), (1.0, 5.0), (1.0, nan)]
print(list(df.groupby(['col1', 'col2'], dropna=False, sort=False).groups.keys()))
# But this is is the correct order, with keys appearing in the order they were in in the original dataframe:
# [(nan, 4.0), (1.0, 5.0), (1.0, nan), (0.0, 7.0), (0.0, 5.0)]
print(list(df.groupby(['col1', 'col2'], dropna=False, sort=False).sum().index))Issue Description
The sort order doesn't match the order for other groupby aggregations whether sort=False or sort=True.
Expected Behavior
The sort order should match the order for other groupby aggregations whether sort=False or sort=True, i.e. sort by key columns when sort=True, and sort by original order in dataframe when sort=False.
Installed Versions
INSTALLED VERSIONS
commit : a671b5a
python : 3.9.18.final.0
python-bits : 64
OS : Darwin
OS-release : 23.2.0
Version : Darwin Kernel Version 23.2.0: Wed Nov 15 21:55:06 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6020
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 2.1.4
numpy : 1.26.3
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 68.2.2
pip : 23.3.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 8.18.1
pandas_datareader : None
bs4 : None
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.4
qtpy : None
pyqt5 : None