API/DES: Non-Nanosecond Tracker

[jbrockmendel]

Support for non-nanosecond timedelta64, datetime64, and datetime64tz is coming along. The next big planned steps are to get the Timedelta and Timestamp scalars to support non-nano resolutions. There are a few design options that I'd like to get input on cc @pandas-dev/pandas-core

I'm doing Timedelta first mostly because that is more conducive to doing as a scoped PR with dedicated testing. Current plan is to make a dedicated constructor like Timedelta._from_value_and_reso (so i can write tests) that can can be removed once we decide on the public behavior. Which brings us to the questions:

• Do we add a reso-like keyword to the constructors? Or use/respect "unit"?
• With non-nano np.timedelta64 objects that don't overflow when cast to nano, do we still cast? e.g. does `np.datetime64(4, "s") become ns or stay s?
• Similar with pytimedelta. pd.Timedelta(timedelta(days=106752)) currently raises, in the future will presumably come back with a 'us' reso. So what about currently non-raising cases like pd.Timedelta(timedelta(days=106751))? Does it stay ns or become us?

Other

1. What happens to (class attributes) (Timestamp|Timedelta)(min|max|resolution)?

Update 2023-01-16 Remaining TODO list

• BUG: non-nano strftime returns wrong results #50787
• BUG: Timestamp.replace handle out-of-pydatetime range #50348 necessary for date_range with e.g. Year/Quarter offsets
• BUG: Index.get_indexer with mixed-reso datetime64s #50690
• API: Timestamp and Timedelta .value changing in 2.0 #49076
• API: Timestamp/Timedelta resolution in __repr__ #49059
• BUG: DatetimeIndex with non-nano values and freq='D' throws ValueError #50727
• BUG: resample don't work with non-nanosecond reso #50656
• BUG: merge_asof with non-nano #50835
• BUG/ENH: groupby.quantile support non-nano #50978
• preserve non-nano in unique (xref ENH: unique/factorize preserve non-nano #48670, BUG/API: preserve non-nano in factorize/unique #51978)
• handle non-nano in infer_dtype
• handle non-nano in maybe_convert_objects
• handle non-nano in array_to_datetime
• ENH: add unit, as_unit to dt accessor #51223
• BUG: timedelta64[s] series constructor isn't equal with alternative constructor using to_timedelta unit='s' #48312
• API: Series[td64].astype("timedelta64[unsupported]") #48979
• ENH: Timedelta cannot span more than 293 years => implementation limitation #35687
• BUG: pd.Timedelta breaks hash invariant #44504
• Timestamp unit is not respected #27490
• Support or default to less detailed datetime64 #7307
• Can we de-duplicate any of NPY_DATETIMEUNIT, FreqGroup, Resolution, PeriodDtypeCode (I expect all the low-hanging fruit has been gotten, but worth trying)
• change how we interpret ints in e.g. pd.Series([1, 2, 3], dtype="m8[s]") to match numpy?
• pd.Timedelta(days=365*1000) raises
• BUG: comparing pd.Timedelta with timedelta.max fails #49021
• Period overflows with dates > pd.Timestamp.max 2262-04-11 #28104

[mroeschke]

Quick thoughts:

Do we add a reso-like keyword to the constructors? Or use/respect "unit"?

My gut is saying this should use/respect "unit" as it's not clear to me how "unit" and a reso-like keyword would cooperate?

With non-nano np.timedelta64 objects that don't overflow when cast to nano, do we still cast?

Based on the GH issues I remember seeing in the past, generally people were hoping datetime64[s] was not cast to datetime64[ns] for example, but this behavior has been around for a while and some users may have gotten use to it. I would think this would be a welcome change as the "old" behavior could be preserved with astype("...64[ns]")?

Similar with pytimedelta...

Under the assumption unit : str, default 'ns' is still the default after this feature, I would expect "ns" for the current non-raising cases

[rhshadrach]

Do keep in mind I don't have much experience using the date functionality even as-is.

Do we add a reso-like keyword to the constructors? Or use/respect "unit"?

For the "use/respect unit" option, I think this would mean that a user specifying pd.Timedelta(1, unit='days') would always have a resolution of days. Is there a downside to this? One thought is combining different units (and hence, resolutions) would constantly require conversions. But maybe different resolutions would be cached under the hood.

Another option, if being able to specify the resolution is advantageous, would be to allow units such as days[s] or days[ns] etc. Unspecified could then default to ns.

pd.Timedelta(timedelta(days=106752)) currently raises, in the future will presumably come back with a 'us' reso.

Is this value-dependent behavior? If everything "just works", then I don't have much of a concern. However, if getting back one resolution vs another can cause odd results in edge cases, this could be surprising and lead to hard to find bugs. If that is the case, I would lean toward raising and making the user specify the resolution.

[jbrockmendel]

For the "use/respect unit" option, I think this would mean that a user specifying pd.Timedelta(1, unit='days') would always have a resolution of days. Is there a downside to this?

Not really, just an API change.

One thought is combining different units (and hence, resolutions) would constantly require conversions.

Yah, I've handled comparisons between scalars of different resolutions, but addition/subtraction (and division for Timedelta) it's not obvious what the right behavior is (i think numpy silently converts in ways that sometimes overflow, have examples/tests written down somewhere). I think we provide an .astype-like API to allow conversion and otherwise raise on potentially-ambiguous operations.

[jorisvandenbossche]

• Do we add a reso-like keyword to the constructors? Or use/respect "unit"?

The existing keyword is about "how to interpret the user input", and what we now want is "how to store the parsed value". Those two are not necessarily the same, and I personally don't think we can simply assume they are (e.g. I think it would be quite common use case that you want to convert values that represent for example hours, but you want to store then in microseconds for compatibility with other data).

In addition, we allow more units here for this keyword than what I would implement as possible resolution of the dtype.
But given the responses above, we should maybe have this discussion explicitly about which resolutions we want to support. For example, about:

For the "use/respect unit" option, I think this would mean that a user specifying pd.Timedelta(1, unit='days') would always have a resolution of days. Is there a downside to this?

My answer would be: we don't support a resolution of days, so such a resulting value is simply not possible (and thus it would be impossible to always respect the unit from the constructor.

(it's certainly unfortunate, though, that the unit keyword is not directly available for specifying the resulting storage unit .., as that would be the logical keyword name for this)

• Similar with pytimedelta. pd.Timedelta(timedelta(days=106752)) currently raises, in the future will presumably come back with a 'us' reso. So what about currently non-raising cases like pd.Timedelta(timedelta(days=106751))? Does it stay ns or become us?

I think long term, the logical behaviour would be that any datetime.timedelta conversion by default results in "us" (since that is the "native" resolution of datetime.timedelta), regardless of the actual value.
That will be a breaking change though, and I suppose we can only make that at some point in a major release (eg when also changing the default resolution), and on the short term keep using "ns" for this (and thus value-dependent behaviour).

[jbrockmendel]

I'm reaching a point where making further progress depends on having non-nano DatetimeTZDtype. That is easy to implement, but would be an API change bc ATM passing non-nano to the DatetimeTZDtype constructor raises (which we test).

Selfishly, it would be very convenient to break the no-changes-in-minor-versions rule and do this immediately. Since this is a pretty tiny corner case, I doubt anyone would complain. But the policy is there for a reason.

Thoughts on a) whether this is reasonable or b) viable alternatives? cc @mroeschke @jreback @jorisvandenbossche

[mroeschke]

ATM passing non-nano to the DatetimeTZDtype constructor raises (which we test).

Does it raise with a not-implemented-like error message? If so, I think the "API change" could be advertised as an "enhancement"

[jbrockmendel]

ValueError

[datapythonista]

If I'm not missing anything, the only user code that could "break" with what you propose is if the user has code creating second precision datetime data, which currently breaks, and after your changes would work.

Personally I think:

1. Is very unlikely that users have code with second precision, since it's not implemented. Or precision obtained at runtime
2. If someone implemented this code, I assume would be to use it if this was ever implemented, and would be happy to see it finally working

If I'm not missing something, I'd personally add this directly without overcomplicating things.