DEPR: Deprecate returning a DataFrame in Series.apply

[topper-123]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Series.apply (more precisely SeriesApply.apply_standard) normally returns a Series, if supplied a callable, but if the callable returns a Series, it converts the returned array of Series into a Dataframe and returns that.

>>> small_ser = pd.Series(range(3))
>>> small_ser.apply(lambda x: pd.Series({"x": x, "x**2": x**2}))
   x  x**2
0  1     1
1  2     4
2  3     9

This approach is exceptionally slow and should be generally avoided. For example:

import pandas as pd
>>> ser = pd.Series(range(10_000))
>>> %timeit ser.apply(lambda x: pd.Series({"x": x, "x**2": x ** 2}))
658 ms ± 623 µs per loop

The result from the above method can in almost all cases be gotten faster by using other methods. For example can the above example be written much faster as:

>>> %timeit ser.pipe(lambda x: pd.DataFrame({"x": x, "x**2": x**2}))
80.6 µs ± 682 ns per loop

In the rare case where a fast approach isn't possible, users should just construct the DataFrame manually from the list of Series instead of relying on the Series.apply method.

Also, allowing this construct complicates the return type of Series.Apply.apply_standard and therefore the apply method. By removing the option to return a DataFrame, the signature of SeriesApply.apply_standard will be simplified into simply Series.

All in all, this behavior is slow and makes things complicated without any benefit, so I propose to deprecate returning a DataFrame from SeriesApply.apply_standard.

[HarrisonWilde]

Could I ask, in the following situation, what should I do instead of .apply(pd.Series):

    train_configs = experiments["train_config"].apply(pd.Series)
    model_configs = experiments["model_config"].apply(pd.Series)
    experiments = experiments.drop(columns=["train_config", "model_config"]).join(train_configs).join(model_configs)

[topper-123]

You don’t show the contents of those columns, nor the intended output, so difficult to help. However, from the looks of it, you should be using ‘ experiments["train_config"].map(pd.Series)’ etc.

If not, you need to give more information anout data and intended output, so I can better assess the need.

[HarrisonWilde]

Apologies, I meant to attach the message below too:

When I run the code above, I see these warnings:

...: FutureWarning: Returning a DataFrame from Series.apply when the supplied function returns a Series is deprecated and will be removed in a future version.
  train_configs = experiments["train_config"].apply(pd.Series)
...: FutureWarning: Returning a DataFrame from Series.apply when the supplied function returns a Series is deprecated and will be removed in a future version.
  model_configs = experiments["model_config"].apply(pd.Series)

This can be reproduced with version 2.1.0 of pandas and a dataframe of the following form:

experiments = pd.DataFrame([{"train_config": {"num_epochs": 30, "patience": 10}, "model_config": {"num_layers": 3, "layer_dim": 256}}])

A lot of the discourse online implies this is the "standard" approach to unpacking dictionaries in columns. I am now using this method, but wondered if there was a recommended workflow in light of the deprecation:

    experiments = experiments.join(
        pd.DataFrame(experiments.pop("train_config").values.tolist(), index=experiments.index)
    )
    experiments = experiments.join(
        pd.DataFrame(experiments.pop("model_config").values.tolist(), index=experiments.index)
    )

[topper-123]

The way I'd do it:

>>> import pandas as pd
>>> experiments = pd.DataFrame([{
    "train_config": {"num_epochs": 30, "patience": 10}, 
    "model_config": {"num_layers": 3, "layer_dim": 256},
}])
>>> train_config = pd.concat(experiments["train_config"].map(pd.Series).values)

and something similar to model_config. Also you may want to do pd.concat(..., axis=1) if that's the output shape you're after.

[rhshadrach]

@HarrisonWilde - can your input only be a single row? I get odd results if I try your method on multiple rows.