Description
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
(optional) I have confirmed this bug exists on the master branch of pandas.
Code Sample, a copy-pastable example
import pandas as pd
from datetime import datetime
df = pd.DataFrame({"OOB": [datetime(1,1,1), datetime(9999,9,9)]})
Actual output
>>> df["OOB"].clip(lower=pd.Timestamp.min, upper=pd.Timestamp.max)
0 2262-04-11 23:47:16.854775807
1 1677-09-21 00:12:43.145224193
Name: OOB, dtype: object
Problem description
The problem is that the datetime(1,1,1)
ended up being clipped to pd.Timestamp.max
i.e the lowest value became the highest and vice-versa.
I suspect that this is to do with the interaction between datetime objects and Timestamps. The last version where this was functioning as I expected was 1.2.5
.
Notes:
- I couldn't use the equivalent datetime to pd.Timestamp.{min,max} as the lower and upper bound because even then I would get an OOB error when trying to
pd.to_timestamp
theSeries
. Obviously there are kind of gross ways to end up with aSeries
that has been clipped to min/max.
Expected Output
This is what I expected to happen (and what happened in previous versions.
>>> df["OOB"].clip(lower=pd.Timestamp.min, upper=pd.Timestamp.max)
0 1677-09-21 00:12:43.145224193
1 2262-04-11 23:47:16.854775807
Name: OOB, dtype: object
Here is what happens if I clip using datetimes...
>>> df["OOB"].clip(lower=pd.Timestamp.min.to_pydatetime(), upper=pd.Timestamp.max.to_pydatetime())
sys:1: UserWarning: Discarding nonzero nanoseconds in conversion
0 1677-09-21 00:12:43.145224
1 2262-04-11 23:47:16.854775
I get the expected output this way however, I fail in my ultimate goal of obtaining a clipped Series[Timestamp]
and just get a Series[datetime]
.
>>> pd.to_datetime(df["OOB"].clip(lower=pd.Timestamp.min.to_pydatetime(), upper=pd.Timestamp.max.to_pydatetime()))
...
pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 1677-09-21 00:12:43
Output of pd.show_versions()
INSTALLED VERSIONS
commit : c7f7443
python : 3.8.10.final.0
python-bits : 64
OS : Darwin
OS-release : 20.1.0
Version : Darwin Kernel Version 20.1.0: Sat Oct 31 00:07:11 PDT 2020; root:xnu-7195.50.7~2/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_AU.UTF-8
LOCALE : en_AU.UTF-8
pandas : 1.3.1
numpy : 1.20.3
pytz : 2021.1
dateutil : 2.8.1
pip : 21.2.1
setuptools : 49.6.0.post20210108
Cython : None
pytest : None
hypothesis : None
sphinx : 3.4.3
blosc : None
feather : None
xlsxwriter : 1.4.3
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.8.6 (dt dec pq3 ext lo64)
jinja2 : 3.0.1
IPython : 7.24.1
pandas_datareader: None
bs4 : None
bottleneck : 1.3.2
fsspec : 0.8.7
fastparquet : None
gcsfs : None
matplotlib : 3.4.2
numexpr : 2.7.3
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 4.0.0
pyxlsb : None
s3fs : None
scipy : 1.6.3
sqlalchemy : 1.4.17
tables : None
tabulate : 0.8.9
xarray : 0.18.2
xlrd : None
xlwt : None
numba : 0.53.1