BUG: Inconsistent behaviour of apply with result_type='expand' for functions that can return NaN type values

[finnhacks42]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
print(pd.__version__)

def split_string(s):
    if pd.isnull(s):
        return None
    parts = s.split(',')
    return {'X':parts[0],'Y':parts[1]}

df1 = pd.DataFrame({'A':['1,2',None,'5,7']})
df2 = pd.DataFrame({'A':[None,'1,2','5,7']})

# this gives a DataFrame with columns X and Y
out1 = df1.apply(lambda row:split_string(row["A"]), axis="columns", result_type="expand")

# this gives a Series
out2 = df2.apply(lambda row:split_string(row["A"]), axis="columns", result_type="expand")

print(type(out1))
print(type(out2))

Issue Description

If apply with result_type='expand' is passed a function that can return NaN type values, the type of the output depends on whether the first return from the function is null. If the first value is not NaN, apply will return a DataFrame as expected. However, if the first value is NaN apply will return a Series.

Expected Behavior

I am not sure what the expected behavior is here. Ideally, the results of apply on the two DataFrames should give an output with the same type. If this is not possible - a warning could be thrown to indicate that returning NaN type values with result_type='expand' can lead to unexpected results.

Installed Versions

INSTALLED VERSIONS

commit : 0691c5c
python : 3.9.21
python-bits : 64
OS : Linux
OS-release : 6.9.3-76060903-generic
Version : #202405300957173214176822.04~f2697e1 SMP PREEMPT_DYNAMIC Wed N
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_AU.UTF-8
LOCALE : en_AU.UTF-8

pandas : 2.2.3
numpy : 2.0.2
pytz : 2024.1
dateutil : 2.9.0.post0
pip : 25.0.1
Cython : None
sphinx : None
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : None
lxml.etree : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : None
pyreadstat : None
pytest : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2025.1
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report! apply performs inference from the first value it gets. If this is not list-like, the result is inferred to not be list-like and therefore expand has no effect. You can instead do:

def split_string(s):
    if pd.isnull(s):
        return {}
    parts = s.split(',')
    return {'X':parts[0],'Y':parts[1]}

I suppose we could add a note in the docstring mentioning that inference is done by looking at the first result. PRs to add this are welcome!

[PenguinPen]

take