Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
df1 = pd.DataFrame(
{
"key": ["a", "c", "e", "a", "c", "e"],
"lvalue": [1, 2, 3, 1, 2, 3],
"group": ["a", "a", "a", "b", "b", "b"]
}
)
df2 = pd.DataFrame({"key": ["b", "c", "d"], "rvalue": [1, 2, 3]})
# works
pd.merge_ordered(df1, df2, fill_method="ffill", left_by="group",)
# Fails with `pd.merge(df1, df2, left_on="group", right_on='key', how='left').ffill()`
pd.merge_ordered(df1, df2, fill_method="ffill", left_by="group", how='left')
# Inverting the sequence also seems to work just fine
pd.merge_ordered(df2, df1, fill_method="ffill", right_by="group", how='right')
# Fall back to "manual" ffill
pd.merge(df1, df2, left_on="group", right_on='key', how='left').ffill()Issue Description
When using merge_ordered (as described in it's documentation page), this works with the default (outer) join - however fails when using how="left" with the following error:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[1], line 14
12 pd.merge_ordered(df1, df2, fill_method="ffill", left_by="group",)
13 # Fails with `pd.merge(df1, df2, left_on="group", right_on='key', how='left').ffill()`
---> 14 pd.merge_ordered(df1, df2, fill_method="ffill", left_by="group", how='left')
File ~/.pyenv/versions/3.11.4/envs/trade_3114/lib/python3.11/site-packages/pandas/core/reshape/merge.py:425, in merg
e_ordered(left, right, on, left_on, right_on, left_by, right_by, fill_method, suffixes, how)
423 if len(check) != 0:
424 raise KeyError(f"{check} not found in left columns")
--> 425 result, _ = _groupby_and_merge(left_by, left, right, lambda x, y: _merger(x, y))
426 elif right_by is not None:
427 if isinstance(right_by, str):
File ~/.pyenv/versions/3.11.4/envs/trade_3114/lib/python3.11/site-packages/pandas/core/reshape/merge.py:282, in _gro
upby_and_merge(by, left, right, merge_pieces)
279 pieces.append(merged)
280 continue
--> 282 merged = merge_pieces(lhs, rhs)
284 # make sure join keys are in the merged
285 # TODO, should merge_pieces do this?
286 merged[by] = key
File ~/.pyenv/versions/3.11.4/envs/trade_3114/lib/python3.11/site-packages/pandas/core/reshape/merge.py:425, in merg
e_ordered.<locals>.<lambda>(x, y)
423 if len(check) != 0:
424 raise KeyError(f"{check} not found in left columns")
--> 425 result, _ = _groupby_and_merge(left_by, left, right, lambda x, y: _merger(x, y))
426 elif right_by is not None:
427 if isinstance(right_by, str):
File ~/.pyenv/versions/3.11.4/envs/trade_3114/lib/python3.11/site-packages/pandas/core/reshape/merge.py:415, in merg
e_ordered.<locals>._merger(x, y)
403 def _merger(x, y) -> DataFrame:
404 # perform the ordered merge operation
405 op = _OrderedMerge(
406 x,
407 y,
(...)
413 how=how,
414 )
--> 415 return op.get_result()
File ~/.pyenv/versions/3.11.4/envs/trade_3114/lib/python3.11/site-packages/pandas/core/reshape/merge.py:1933, in _Or
deredMerge.get_result(self, copy)
1931 if self.fill_method == "ffill":
1932 if left_indexer is None:
-> 1933 raise TypeError("left_indexer cannot be None")
1934 left_indexer = cast("npt.NDArray[np.intp]", left_indexer)
1935 right_indexer = cast("npt.NDArray[np.intp]", right_indexer)
TypeError: left_indexer cannot be None
This did work in 2.1.4 - and appears to be a regression.
A "similar" approach using pd.merge, followed by ffill() directly does seem to work fine - but this has some performance drawbacks.
Aside from that - using a right join (with inverted dataframe sequences) also works fine ... which is kinda odd, as it yields the same result, but with switched column sequence (screenshot taken with 2.1.4).
Expected Behavior
merge_ordered works as expected, as it did in previous versions
Installed Versions
INSTALLED VERSIONS
commit : fd3f571
python : 3.11.4.final.0
python-bits : 64
OS : Linux
OS-release : 6.7.0-arch3-1
Version : #1 SMP PREEMPT_DYNAMIC Sat, 13 Jan 2024 14:37:14 +0000
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 2.2.0
numpy : 1.26.3
pytz : 2023.3
dateutil : 2.8.2
setuptools : 69.0.2
pip : 23.3.2
Cython : 0.29.35
pytest : 7.4.4
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.3
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.3
IPython : 8.14.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.2
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2023.12.1
gcsfs : None
matplotlib : 3.7.1
numba : None
numexpr : 2.8.4
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 15.0.0
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.12.0
sqlalchemy : 2.0.25
tables : 3.9.1
tabulate : 0.9.0
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None