GH6848 silently changed series.sort from stable to unstable sort

[armaganthis3]

#6848 is no where mentioned in the release notes. There's a single item on .order

changing to quicksort, but it doesn't explicitly warn that the sort was previously stable
and now isn't. There's also no warning about series.sort changing as well.
That API has been around forever and afaict nothing was gained by breaking it.

This was a fairly reckless change in my opinion, I do wish the current maintainers
cared more about backwards-compatibility then they seem to in the latest releases.

[jreback]

http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#v0-14-0-may-31-2014

about 1/2 down in API changes. not sure where you are looking.

backwards compat is cared about WAY too much IMHO actually.

[jreback]

@armaganthis3 you are free of course to comment on any and all issues. This change was to conform a completely broken API that was different for inplace (sort) and a copy (order). IMHO that has been a huge source of confusion for quite a while.

Their are always a number of API changes in major versions; you are encouraged to read them. Not every change can warrant a big warning (and IMHO the warnings are ignored just as much as the whatsnew).

[armaganthis3]

Pandas has more breaking changes per version, those releases are much more frequent
compared to other pydata packages and unlike others pandas does not maintain a stable branch.

Considering that, I think you should care more about backwards-compatibility not less.
It's terrible that you don't think this stuff matters. That you actually think APIs are not
being broken fast enough is... well.

Just one example:
#5027 (comment)

[jreback]

pandas breaks on big version number. and those are all very carefully though out. and very well documented. If you didn't read about this change, then whom are you blaming?

I am not sure what 'does not maintain a stable branch means'. you can install any version you want. it is your choice to upgrade or not. you do not have to upgrade at all if you don't want.

I am not sure where you get the statement about backwards compatibility.

I personally care about back compat quite a lot. Have you seen the tremendous lengths pandas has gone to maintain backwards compat with numpy 1.6 and pickle compat < 0.12!!!!!

This takes quite a bit of time, and pandas does not have the resources as other pydata projects, yet still seems to produce very high quality s/w with many many contributors.

I think you are completely misreading my comments. pls read them again.

[armaganthis3]

Pandas breaks only on major version numbers, ok, but it also releases a new major version
every 3 months. twice you've claimed I haven't read the release notes - you might
read my original comment again, you might have misread that.

"does not maintain a stable branch" means what it says: there's no long term stable branch.
bug fixes are not back-ported, perf regressions are not back-ported. perf regressions are common
and bugs are common (e.g 0.13), to get them fixed, upgrading is the only option pandas offers.

Your view of backwards compat is based on your words:

backwards compat is cared about WAY too much IMHO actually.

It's funny you should mention those two example for maintaing back-compat. First because
back-compat in this case has nothing to do with what version of numpy you support, the OP is about pandas code remaining compatible with pandas, and secondly because you're about
to throw away numpy 1.6 in another issue (#7711)
even though it's still part of some LTS distros, for no good reason, and pickle compat was
broken by 0.14 see #7748, where you described it as:

not a bug per-se, more of a slightly change in the compat of frequencies in terms of
backward-compat for pickle

Which is perhaps the worst double-speak I've come across outside fox news.

As for having no resources (but at the same time, having "many many contributors")
sure ok, I'd expect that to lead to slower development, not accelerated breakage,
but nevermind point taken. I'm not here to assign grades anyway - not my role, not my right.

Very simply, as a user who maintains a prod system that relies on pandas I care
muchly about back-compat. I would like pandas code I write today to work a year from now.
It doesn't seem like that much to expect nor does it seem to be much of an issue with other
projects I rely on.

[jreback]

@armaganthis3 talk about double-speak!

I read your original comment. The change that you brought up WAS in the whatsnew. I guess you STILL did not read it. Yes its a one line change, for a VERY MINOR change. WHICH was documented. and provided consistency between a very confusing function. I guess you are annoyed that you simply didn't read the whatsnew are now trying to 'blame' someone else. go figure.

you want back ported fixes/perf regressions. great! I would like to see some contributions then.

pandas development has always been done this way (I think prob because of limited resources; maintaing a 'stable' branch is VERY time consuming).

and to be frank, pandas has introduced many new features over time. things break. things get fixed. pandas is an active project.

[jreback]

@armaganthis3 you point out #7711. do you have any idea what back-compatilbility even means? what a PITA maintaining 1.6 IS????? great. LTS still has an old numpy. fantastic, but so WHAT? the system installs are pretty much useless for scientific python, if you would actually read my comments in that issue. And it has been brought up since 0.11 IIRC. This has been delayed quite a long time. Talk about back compat!

That is what I am talking about in terms of time spent. If we didn't have to maintiain this 1.6 compat then I suspect a fair number of new features could be introduced.

Yes pickle compat breaks occasionally. We went to VERY great lengths to preserve this. but again if would actually read #7748, that actually its a very recent bug fix in 0.14.1 to provide COMPAT across frequencies. Did happen to have a test for this, so it broke. it will get fixed. So instead of 6000 odd tests, we will have 6001.

How many projects have this level of care about back-compat????

[jreback]

@armaganthis3 my last point. Would you rather have a project release every 3-4 months or debate everything endlessly and have it release once a year. More frequent releases allow for more community uptake, so are you saying that is a BAD thing?

[jorisvandenbossche]

@armaganthis3 @jreback

@armaganthis3 You raise good points, and it always interesting to get the experience from another viewpoint. Pandas has maybe the 'bad luck' of already being very popular but still rather young and rapidly changing, and a good balance between both is not always easy.
But @jreback also has a point that we already care a lot about this. Hard work that is maybe not always directly visible. But can it better? Probably, yes. We are also human, and things can always be improved. Getting constructive feedback from users like you is essential for that.

Backwards compatibility is an important issue. As the heating of this discussion also shows! It shows that it is something both of you care about. But a heated discussion is most of the time not very constructive. So maybe we should let this cool down for a moment. Please do that, both of you.
And maybe we can continue it later, in a constructive way, to look at concrete points that could be improved. As that is what we all want, to improve pandas!

[armaganthis3]

Here we are 2 days later, and hopefully calm. Excessive punctuation neatly tucked away I hope.
Let's pick this up, It's important.

@jreback made a lot of comments, I'll try to cover most of his points, skipping over the
nasty provoking bits and try to focus back on the issue I actually care about.

Personally, I don't need numpy 1.6 support. Others might or might not. The essential issues
I'm raising is not numpy support but the stability of pandas APIs, that is, keeping pandas code working across versions. I found it ironic that that the example jreback chose to give is what's
being eliminated in the next version, but that's a digression.

Keeping pandas code working across pandas versions is what back-compatible means,
it's what I care about, It's what I'm worried about.

jreback says that bugs and regressions are inevitable and in that a least he is absolutely right.
They are also by definition unintentional so It's an entirely different thing when
maintainers break an API, knowingly and deliberately. It's not an "unavoidable mistake"
it's a concious choice.

You may claim this breakages are "well-thought out", from what I've seen
the reason is not an unavoidable breakage due to evolving code, but misguided
aesthetics (toted as "consistency" or "cleanliness" or "nice") being masquaraded
as an "overpowering" reason to break established API and the code that uses them.

"So yur saying frequet release is badd ?????" jreback asks. Not at all.
You could have daily releases - no one is forcing you to break APIs while doing it.
The problem is not the release scheduele: it's the combination of having frequent
major releases, having no maintenance branch and having a liberal attitude about
breaking APIs. It's The combination of all these.

"WELL, are YOU going to do the VERY time consuming work involved in keeping a
maintenance branch????" jreback might say (duly ignoring the combination point I made).
You're right, I'm not. Though pandas never had a maintenance afaik and how time-consuming
it is not known, but let's suppose it is...

Well, how about simply not break APIs so liberally? Use stricter criteria for allowing
breakage, make legacy code support higher priority then it is. Do not break existing
code deliberately. That would solve the problem without requiring more resources or
changes to the release schedule.

Would it help if users said please-oh-please don't break APIs?

Finally, about the point that pandas is young, I'll note that pandas is 3 years old, has (10s of) thousands of users in commercial and science application, Is used in production. Is on it's
2nd or 3rd generation of developers. There are lots of SO questions and existing tutorials.
Lots of existing code. It has great documentation. In fact It's mature by all criteria I can
think off. all except API stability that is.

[armaganthis3]

I'll add another example of how back-compat is often dismissed lightly, this is from today,
It's quite easy to find them even if you're not looking: #7329

Another issue with legacy pickles, which @jreback so forcefully claimed he cares about.

@Kermit666 said:

I created it in Pandas 0.13.1 and I can read it normally with that version. (but not 0.14.1)

@jreback's response:

this issue is not fixable in that older pickles that are prior to this fix are not unpickle able

pandas 0.13.1 can pickle and unpickle a data file, and 0.14.1 can't. according
to jreback, that's not a problem. it's "minor" perhaps that users can't open their data
after upgrading pandas, and it's not "fixable".

By the way, introduced in 0.14.1, not a major release. care indeed.

[cpcloud]

@armaganthis3 The pickle breakage seems like a regression to me. While I lean towards the position of moving fast and upgrading only if you've you read the release notes thoroughly, breaking IO across a single (major) version number is moving too fast. Pickles suck, but people use them and we support them so we should try really hard not to break IO related code IMHO.

In the case of sort vs. order I disagree. The release notes are there, and you don't have to upgrade if you don't want to. IMO it is more consistent if you have them either both stable or both unstable, but not different.

@armaganthis3 Would like you to make a list of the API changes you find especially egregious? That would help us evaluate whether there are unintended regressions or intentional API breakage.

I agree that this in an important issue, but I don't think it's as simple as "preserve back-compatibility at all costs".

[jreback]

@armaganthis3 you are excellent at taking things out of context / not reading the entire issue.

Would it help if users said please-oh-please don't break APIs?

This feature addition for 0.15.0, #7217 (comment), does NOT BREAK ANY API's at all.

So why are you bringing up an point that is completely irrelevant?

and you point at #7329, which was warned right here: http://pandas.pydata.org/pandas-docs/version/0.14.0/whatsnew.html, and turns out is a regression, that should have gotten caught in 0.14.0, but unfortunately our 6000 tests didn't help. Well a fix going forward, as well as a work-around was put in place. I'd like to catch these things on the first go around, but pandas is a big project and not everything can be caught.