RLS: 0.24.0

[jreback]

Tracking issue

Open PRs
Open Issues

Let's whittle down. I just looked the lastest whatsnew and its huge. Let's get this out sooner rather than later. I know there are some blocking issues: DatetimeArray & What do do with CalendarDay.

[jbrockmendel]

The idea is for 0.24.0 to be the last py2-supporting release, right?

#24021 fixes a corner case behavior in Timestamp comparisons, but also introduces an inconsistency between py2/py3 behavior. #21394 made the analogous change to Timedelta comparisons.

The least consistent thing we could do is to keep the status quo, changing Timedelta behavior but not Timestamp behavior. The question is whether to a) merge #24021 and have mismatched py2/py3 behavior in 0.24.0, or b) revert #21394 until after 0.24.0 and wait to change both in the first py3-only release.

I lean slightly towards option b.

[TomAugspurger]

The idea is for 0.24.0 to be the last py2-supporting release, right?

Basically, though I assume we'll do some backporting and do a 0.24.1 or 2 that supports py2 as well.

I'm not super-familiar with this section, but your option b sounds reasonable.

Although... I could live with inconsistent py2 / py3 behavior. And would we be consistent with the building That wouldn't be the only one.

Question: with
#24021 would we be consistent with the builtin timedelta of each version?

[h-vetinari]

This release is indeed big, but v.0.24 is also quite special because it will effectively define the 1.0 API (in the sense of the no-deprecation policy between 0.24 and 1.0), and also because of the magnitude of the whole EA effort obviously.

But - despite a lot of hard work - the current state still feels a bit half-baked:

• The Datetime refactor: https://github.com/pandas-dev/pandas/projects/4#card-15359769
• ExtensionArray: https://github.com/pandas-dev/pandas/issues?q=is%3Aissue+is%3Aopen+label%3AExtensionArray, including several points touching core functionality like __setitem_ (Allow dtype promotion in Series[ExtensionArray].__setitem__? #24020) or merge (BUG: merging on an Integer EA raises #23020) or interaction between EAs and other (pandas) objects (IntegerArray + List returns ndarray #22962 / TST: Add ExtensionArray base tests for ops between pandas objects and EAs #22974), groupby (Add interface for defining an ExtensionIndex #22861) etc.
• regressions: https://github.com/pandas-dev/pandas/issues?q=is%3Aopen+is%3Aissue+label%3ARegression
• pd.Array: API: added array #23581
• consistency in new attributes (recommending .array over .values everywhere?) Public Data Followups #23995
• For the sake of consistency (especially in relation to EAs as well), I think one should really clean up the API around unique before 1.0 (API/ENH/DEPR: Series.unique returns Series #24108)
• several other things I'm leaving out for brevity

Realistically, a release before the end of the year would mean a cut-off in ~10 days max, which seems unrealistic from my POV, even if cutting corners.

Given that the following statement from @TomAugspurger above

Basically, though I assume we'll do some backporting and do a 0.24.1 or 2 that supports py2 as well.

effectively means more PY2 support at the beginning of 2019 anyway, I think one should consider not trying to force the release before year end.

If there is a release before the end of the year (resp. before the most important issues are sorted out), then I especially agree with Tom that there will need to be a 0.24.1 for PY2, as 0.24.0 will have too many issues (that hopefully surface in the RC, but well...) to be the last release, IMO.

Alternatively (which is simultaneously more in line with https://python3statement.org/, but also more controversial), one could consider to have a last PY2-supporting 0.23.5 this year, and then do 0.24.0 as PY3-only next year...?

[jreback]

@h-vetinari pandas is a volunteer project almost completely. Thus project priorities are set by community consensus and worked towards. We have regular releases in time; 0.24.0 is actually overdue by a few months. Trying to add additional things which themselves need discussion is not going to happen.

Whatever exists in he 0.24.x series is the last release for Python 2, this has long been announced. This is just how it is.

[TomAugspurger]

I don't follow the point of
#24060 (comment). Could you try restating / summarizing it?

I think Py2 vs. Py3 is irrelevant for 0.24.0.

The EA-releated issues you linked are I think all going in 0.24 (I didn't check them all). That's basically the blocker at this point, but I haven't reviewed the backlog recently.

I haven't had time to look into unique.

[h-vetinari]

@jreback

@h-vetinari pandas is a volunteer project almost completely. Thus project priorities are set by community consensus and worked towards. We have regular releases in time; 0.24.0 is actually overdue by a few months. Trying to add additional things which themselves need discussion is not going to happen.

I know all that. I'm just saying that being overdue is a bad reason to rush a release, if some of the core changes have not been fully developed yet (talking mostly about EA + regressions).

Whatever exists in he 0.24.x series is the last release for Python 2, this has long been announced. This is just how it is.

I don't know what's been discussed in other channels, but what I saw on GH was that the main decision was 0.24->0.25->1.0. Re:PY2, it has equally been said (and there's a warning to that effect on the whatsnew's) that there won't be PY2 releases after Dec. 31st 2018. Supporting the 0.24.0 series for PY2 is another ~6-8 month of supporting PY2 (as backporting to 0.24-branch would otherwise be very cumbersome). Of course that's a valid choice, but I just wanted to suggest the possibility of leaving PY2 at the very stable 0.23.5 instead.

[h-vetinari]

@TomAugspurger

I don't follow the point of
#24060 (comment). Could you try restating / summarizing it?
I think Py2 vs. Py3 is irrelevant for 0.24.0.

Sorry this was not clear. The main (interrelated) two points are:

• one should not rush 0.24 due to the stated deadline (for supporting PY2) of Dec 31st.
  • listed some issues for that - some of these (and many more related ones I didn't tag) have been pushed to "Contributions Welcome" instead of v.0.24
  • brought up that 0.24 is special due to the API-lock policy until 1.0, and that I'd think this would be a pity for .unique (but I get that I'm a lone voice in the wilderness about that one)
• point out the discrepancy between the warning box ("Starting January 1, 2019, pandas feature releases will support Python 3 only") and supporting the 0.24-series for PY2, along with the suggestion to have a 0.23.5 for PY2

The EA-releated issues you linked are I think all going in 0.24 (I didn't check them all). That's basically the blocker at this point, but I haven't reviewed the backlog recently.

The backlog has been thinned substantially recently, not least by issues being pushed to "Contributions Welcome" (also for EA issues). This goes towards my point of cutting corners to get this out soon.

That being said, I'm pretty new at this game, and I don't doubt that the core devs got the big picture in view and under control - but pointing out an observation can't hurt, I hope.

I haven't had time to look into unique.

Fair enough, dev time is a very limited resoure. I guess I'll have to try to convince everyone of this in post-1.0 land. ;-)

[jreback]

@h-vetinari

I know all that. I'm just saying that being overdue is a bad reason to rush a release, if some of the core changes have not been fully developed yet (talking mostly about EA + regressions).

If we keep adding and adding, then the release just keeps getting delayed forever. I have drawn a line in sand. This is how you get product out the door. Once the DTA lands fully, we will be in a position to release. So this is not that far away. Sure we could do extra work and just say 0.23.5 is the last PY2 release (and of course release it). But its going to be easier to back to a stable branch, which means the 0.24.x series.

There are always things to add in a release, but this one is already the biggest we have ever had. There are inevitable bugs and so better to do sooner rather than later. Thanks for your contributions. It is not possible to get every major API change here. You are exactly right in that dev time is a truly limited resource.

[h-vetinari]

@jreback
Thanks for the response. I understand you want to get this out the door ASAP, which is fair enough, of course.

But its going to be easier to back to a stable branch, which means the 0.24.x series.

Seems I misunderstood that pandas would stop supporting PY2 per Jan 1st 2019... Maybe should adapt that warning box in the whatsnews then ("v.0.24.x will be the last series that supports PY2; starting with v.0.25.0, pandas will be python3-only"...?)

[mroeschke]

RE: CalendarDay progress #22867 (comment)

TODO
Add all the depreciation warnings (I anticipate there being quite a few of these). This needs to be included in the following:

• Day tick arithmetic with other Ticks, Timedeltas, and DatetimeTZ
• DatetimeIndex.shift (tz-aware only)

Migration
The plan is for _Day (formerly CalendarDay) to just replace Day once the prior Day behavior is replaced.

Concern
During the last dev chat, there was interest for 'D' to be compatible freq argument/offset with both Timedelta and Datetime. I don't see a clear way to make this possible without adding a lot of monkeypatching.

Example: timedelta_range(..., freq='D'); to_offset('D') will return _Day in the future and this offset will need to increment a Timedelta, but _Day + Timedelta is an invalid operation.

[jbrockmendel]

Anyone have opinions on the timestamp/timedelta py2/py3 consistency issue?

[jbrockmendel]

A bunch of deprecations are listed as To Be Removed in 1.0; should 0.25.0 take the place of 1.0 for some of those?

[TomAugspurger]

Anyone have opinions on the timestamp/timedelta py2/py3 consistency issue?

Can you summarize that issue? Ideally we would just follow python (whatever version is running) here I think. But I don't think I fully understand the issue.

A bunch of deprecations are listed as To Be Removed in 1.0; should 0.25.0 take the place of 1.0 for some of those?

I think all of them... Need to discuss that though, some may need be pushed.