Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
def f(x):
return x
df1 = pd.DataFrame({"A": [1,2,3]})
df1.name = "df1"
df2 = pd.DataFrame({"A": [1,2,3], "B": [4,5,6]})
df2.name = "df2"
for df in [df1, df2]:
for raw in [True, False]:
for engine in ["python", "numba"]:
try:
df.apply(f, raw=raw, engine=engine)
except Exception as e:
print(f"{df.name}.apply(f, raw={raw}, engine={engine!r}) failed: {str(e)}")
else:
print(f"{df.name}.apply(f, raw={raw}, engine={engine!r}) passed")Issue Description
The above code gives:
df1.apply(f, raw=True, engine='python') passed
df1.apply(f, raw=True, engine='numba') failed: Length of values (3) does not match length of index (1)
df1.apply(f, raw=False, engine='python') passed
df1.apply(f, raw=False, engine='numba') passed
df2.apply(f, raw=True, engine='python') passed
df2.apply(f, raw=True, engine='numba') passed
df2.apply(f, raw=False, engine='python') passed
df2.apply(f, raw=False, engine='numba') passed
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[12], line 1
----> 1 df1.apply(f, raw=True, engine='numba')
File ~/mambaforge/envs/py3.10/lib/python3.10/site-packages/pandas/core/frame.py:10374, in DataFrame.apply(self, func, axis, raw, result_type, args, by_row, engine, engine_kwargs, **kwargs)
10360 from pandas.core.apply import frame_apply
10362 op = frame_apply(
10363 self,
10364 func=func,
(...)
10372 kwargs=kwargs,
10373 )
> 10374 return op.apply().__finalize__(self, method="apply")
File ~/mambaforge/envs/py3.10/lib/python3.10/site-packages/pandas/core/apply.py:914, in FrameApply.apply(self)
912 # raw
913 elif self.raw:
--> 914 return self.apply_raw(engine=self.engine, engine_kwargs=self.engine_kwargs)
916 return self.apply_standard()
File ~/mambaforge/envs/py3.10/lib/python3.10/site-packages/pandas/core/apply.py:1031, in FrameApply.apply_raw(self, engine, engine_kwargs)
1029 return self.obj._constructor(result, index=self.index, columns=self.columns)
1030 else:
-> 1031 return self.obj._constructor_sliced(result, index=self.agg_axis)
File ~/mambaforge/envs/py3.10/lib/python3.10/site-packages/pandas/core/series.py:575, in Series.__init__(self, data, index, dtype, name, copy, fastpath)
573 index = default_index(len(data))
574 elif is_list_like(data):
--> 575 com.require_length_match(data, index)
577 # create/copy the manager
578 if isinstance(data, (SingleBlockManager, SingleArrayManager)):
File ~/mambaforge/envs/py3.10/lib/python3.10/site-packages/pandas/core/common.py:573, in require_length_match(data, index)
569 """
570 Check the length of data matches the length of the index.
571 """
572 if len(data) != len(index):
--> 573 raise ValueError(
574 "Length of values "
575 f"({len(data)}) "
576 "does not match length of index "
577 f"({len(index)})"
578 )
ValueError: Length of values (3) does not match length of index (1)Expected Behavior
The issue happens on single column DataFrame with numba engine. Expected working as well as python engine.
Installed Versions
INSTALLED VERSIONS
commit : 0691c5c
python : 3.10.14
python-bits : 64
OS : Linux
OS-release : 5.15.0-122-generic
Version : #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 2.2.3
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0
pip : 24.0
Cython : 3.0.7
sphinx : 7.3.7
IPython : 8.25.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.6.0
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.4
lxml.etree : None
matplotlib : 3.9.2
numba : 0.60.0
numexpr : 2.10.0
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : 2.9.9
pymysql : 1.4.6
pyarrow : 16.1.0
pyreadstat : None
pytest : 8.2.2
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.14.0
sqlalchemy : 2.0.31
tables : 3.9.2
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : 0.22.0
tzdata : 2024.1
qtpy : 2.4.1
pyqt5 : None