Skip to content

BUG: Series.clip() behaviour is wrong with Series[datetime] comparison to pd.Timestamp.{min,max} #42794

Closed
@elidhu

Description

@elidhu
  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • (optional) I have confirmed this bug exists on the master branch of pandas.


Code Sample, a copy-pastable example

import pandas as pd
from datetime import datetime
df = pd.DataFrame({"OOB": [datetime(1,1,1), datetime(9999,9,9)]})

Actual output

>>> df["OOB"].clip(lower=pd.Timestamp.min, upper=pd.Timestamp.max)
0    2262-04-11 23:47:16.854775807
1    1677-09-21 00:12:43.145224193
Name: OOB, dtype: object

Problem description

The problem is that the datetime(1,1,1) ended up being clipped to pd.Timestamp.max i.e the lowest value became the highest and vice-versa.

I suspect that this is to do with the interaction between datetime objects and Timestamps. The last version where this was functioning as I expected was 1.2.5.

Notes:

  • I couldn't use the equivalent datetime to pd.Timestamp.{min,max} as the lower and upper bound because even then I would get an OOB error when trying to pd.to_timestamp the Series. Obviously there are kind of gross ways to end up with a Series that has been clipped to min/max.

Expected Output

This is what I expected to happen (and what happened in previous versions.

>>> df["OOB"].clip(lower=pd.Timestamp.min, upper=pd.Timestamp.max)
0    1677-09-21 00:12:43.145224193
1    2262-04-11 23:47:16.854775807
Name: OOB, dtype: object

Here is what happens if I clip using datetimes...

>>> df["OOB"].clip(lower=pd.Timestamp.min.to_pydatetime(), upper=pd.Timestamp.max.to_pydatetime())
sys:1: UserWarning: Discarding nonzero nanoseconds in conversion
0    1677-09-21 00:12:43.145224
1    2262-04-11 23:47:16.854775

I get the expected output this way however, I fail in my ultimate goal of obtaining a clipped Series[Timestamp] and just get a Series[datetime].

>>> pd.to_datetime(df["OOB"].clip(lower=pd.Timestamp.min.to_pydatetime(), upper=pd.Timestamp.max.to_pydatetime()))
...
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 1677-09-21 00:12:43

Output of pd.show_versions()

INSTALLED VERSIONS

commit : c7f7443
python : 3.8.10.final.0
python-bits : 64
OS : Darwin
OS-release : 20.1.0
Version : Darwin Kernel Version 20.1.0: Sat Oct 31 00:07:11 PDT 2020; root:xnu-7195.50.7~2/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_AU.UTF-8
LOCALE : en_AU.UTF-8

pandas : 1.3.1
numpy : 1.20.3
pytz : 2021.1
dateutil : 2.8.1
pip : 21.2.1
setuptools : 49.6.0.post20210108
Cython : None
pytest : None
hypothesis : None
sphinx : 3.4.3
blosc : None
feather : None
xlsxwriter : 1.4.3
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.8.6 (dt dec pq3 ext lo64)
jinja2 : 3.0.1
IPython : 7.24.1
pandas_datareader: None
bs4 : None
bottleneck : 1.3.2
fsspec : 0.8.7
fastparquet : None
gcsfs : None
matplotlib : 3.4.2
numexpr : 2.7.3
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 4.0.0
pyxlsb : None
s3fs : None
scipy : 1.6.3
sqlalchemy : 1.4.17
tables : None
tabulate : 0.8.9
xarray : 0.18.2
xlrd : None
xlwt : None
numba : 0.53.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    DatetimeDatetime data dtypeNumeric OperationsArithmetic, Comparison, and Logical operationsRegressionFunctionality that used to work in a prior pandas version

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions