BUG: way to include all columns within a groupby apply

[madelavar12]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

# Sample DataFrame
df = pd.DataFrame({
    "group": ["A", "A", "B", "B"],
    "value": [1, 2, 3, 4],
})

# Function that operates on the whole group (e.g., adds a new column)
def process_group(group_df):
    group_df["value_doubled"] = group_df["value"] * 2
    return group_df

# Trigger the deprecation warning
result = df.groupby("group").apply(process_group)
print(result)


        group  value  value_doubled
group                              
A     0     A      1              2
      1     A      2              4
B     2     B      3              6
      3     B      4              8
C:\Users\e361154\AppData\Local\Temp\1\ipykernel_15728\2443901964.py:15: DeprecationWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  result = df.groupby("group").apply(process_group)

Issue Description

When using groupby().apply() with a function that modifies and returns the entire group DataFrame, a FutureWarning is raised in pandas >= 2.2. This warning notifies users that in pandas 3.0, the default behavior will change: the grouping columns will no longer be included in the data passed to the function unless include_groups=True is explicitly set. To maintain the current behavior and suppress the warning, users must pass include_groups=False.

This affects workflows where the function operates on the full DataFrame per group and expects the group keys to be included in the data automatically, as was the case in earlier pandas versions.

Expected Behavior

The expected behavior is still what I want from the above example. I just don't want that functionality to be lost in pandas 3.0.

Installed Versions

INSTALLED VERSIONS

commit : 0691c5c
python : 3.10.7
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.22631
machine : AMD64
processor : Intel64 Family 6 Model 140 Stepping 1, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 2.2.3
numpy : 2.2.4
pytz : 2025.2
dateutil : 2.9.0.post0
pip : 25.0.1
Cython : None
sphinx : None
IPython : 8.35.0
adbc-driver-postgresql: None
...
zstandard : 0.23.0
tzdata : 2025.2
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report.

1. You can do df.groupby(...)[df.columns].
2. You can access the groups from the index of the result.
3. You can do df.groupby(..., as_index=False) to have the groups be columns instead of index.

In the event that none of these work for you, you can use DataFrameGroupBy.pipe to have your own helper function.

def include_all_columns(gb, *args, **kwargs):
    return gb[gb.obj.columns]

result = df.groupby("group").pipe(include_all_columns).apply(process_group)

More longer term, pandas core developers are positive on adding expressions, similar to those in PySpark and Polars. If that were to happen, then you could do df.groupby(...)[pd.all()].apply(...).

[rhshadrach]

I'll also add, the example in the OP mutates the provided group_df. This is explicitly not supported. If it works in your use-case, great, but there are various ways you can mutate the argument that will break pandas. You should instead make a copy.

def process_group(group_df):
    group_df = group_df.copy()
    group_df["value_doubled"] = group_df["value"] * 2
    return group_df

[madelavar12]

I agree with your sentiment about the mutation in a general sense, but I also see great use cases for the mutation if adding a column without having to make a copy within every group since that could be computationally intense depending on the DataFrame.

Option 1 is what I have currently been doing df.groupby(...)[df.columns] but that just seems clunky. I am more just wondering why this was a reduction in functionality? Why not just keep the option to include groups? I guess ultimately it ends up being the same thing but I am just not sure why it was just fully deprecated.

The issue isn't with the original groupby, it is within the apply that is causing the issue so options 2 and 3 don't seem to work as you are saying they would unless I am just misunderstanding.

Option 2 with a reset_index specifying the level would work but that also seems clunky especially for a multicolumn groupby:

result = df.groupby("group").apply(process_group, include_groups=False).reset_index(level="group")
print(result)

Option 3 just doesn't return the group anymore:

result = df.groupby("group", as_index=False).apply(process_group, include_groups=False)
print(result)

[rhshadrach]

I am more just wondering why this was a reduction in functionality? Why not just keep the option to include groups? I guess ultimately it ends up being the same thing but I am just not sure why it was just fully deprecated.

apply was the only function that operated on the groups (filters include the groups, but don't operate on them), and even then only in certain cases. It was an inconsistency in the API. Supporting this option in just apply means the groupby internals need to track whether the groups are in the supplied DataFrame itself or outside of it, and what do to in each case. This complicates the internals by adding a whole additional state that needs to be tracked.

Option 3 just doesn't return the group anymore:

Ah, indeed. I think what's going on here is that apply infers that your operation is a transform, and so does not include the groups. I've argument that we should enable the behavior of as_index=False in such cases (#49543), but it seems like it won't gain much traction.