Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
# some methods to transform the example DatetimeIndex objects into other index types
def identity(index: pd.DatetimeIndex) -> pd.DatetimeIndex:
return index
def to_int64_index(index: pd.DatetimeIndex) -> pd.Int64Index:
return index.astype(int)
def to_date_object(index: pd.DatetimeIndex) -> pd.Index:
return pd.Index(index.date, dtype=object)
def to_int_object(index: pd.DatetimeIndex) -> pd.Index:
return pd.Index(index.astype(int), dtype=object)
def to_timestamp_object(index: pd.DatetimeIndex) -> pd.Index:
i = pd.Index(index, dtype=object)
assert i.dtype.str == '|O'
return i
@pytest.mark.parametrize('transform1,transform2', [
(to_date_object, to_date_object), # an example of the issue, so test case will fail
(to_int_object, to_int_object), # an example of the issue, so test case will fail
(to_date_object, identity),
(identity, to_date_object),
(to_timestamp_object, to_timestamp_object),
(identity, identity), # i.e. a DatetimeIndex will be used
(to_int64_index, to_int64_index),
])
def test_issue(transform1, transform2):
"""Concatenating objects with object indices has inconsistent default sorting behaviour when compared to other
index types.
An example inconsistent/non-monotonic result is below which happens when using and object index:
0 1
2000-01-01 0.0 0
2001-01-01 1.0 1
1998-01-01 NaN -2
1999-01-01 NaN -1
"""
# index1 is a subset
index1 = transform1(pd.to_datetime(['2000', '2001']))
index2 = transform2(pd.to_datetime(['1998', '1999', '2000', '2001']))
series1 = pd.Series([0, 1], index1)
series2 = pd.Series([-2, -1, 0, 1], index2)
result = pd.concat([series1, series2], axis=1)
assert result.index.is_monotonic_increasingIssue Description
pd.concat([series1, series2], axis=1) does not implicitly sort the index when the indices do not match when the index is pd.Index(dtype='object'), which is a behavioral change that came in somewhere between 0.19.2 and 1.1.5, and is inconsistent with other index types.
This was an unpleasant surprise found in code when concatenating two series with datetime.date indices (like in the first failing test case) after upgrading a codebase.
A simpler way to reproduce:
print(
pd.concat([
pd.Series([0, 1], pd.Index(pd.to_datetime(['2000', '2001']).date, dtype=object)),
pd.Series([-2, -1, 0, 1], pd.Index(pd.to_datetime(['1998', '1999', '2000', '2001']).date, dtype=object)),
], axis=1)
) 0 1
2000-01-01 0.0 0
2001-01-01 1.0 1
1998-01-01 NaN -2
1999-01-01 NaN -1
Expected Behavior
I was expecting the frame resulting frame to be sorted, as is the case for other index types, and as was the case in at least 0.19.2 but has not been the case since at least 1.1.5.
If I had to guess a rationale for the change between versions, it might be due to assuming that objects aren't necessarily sortable, and the code doesn't even attempt it.
Installed Versions
INSTALLED VERSIONS
commit : 2e218d1
python : 3.8.16.final.0
python-bits : 64
OS : Linux
OS-release : 4.15.0-202-generic
Version : #213-Ubuntu SMP Thu Jan 5 19:19:12 UTC 2023
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.5.3
numpy : 1.24.2
pytz : 2022.7.1
dateutil : 2.8.2
setuptools : 57.5.0
pip : 22.0.4
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None