BLD: pandas should generate PEP 440 compliant dev versions

[shoyer]

Doing an inplace install with recent versions of pip (e.g., with pip install -e .) currently produces this warning:
UserWarning: The version specified ('0.15.2-224-geadfd92') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.

We should update setup.py to produce PEP 440 compliant versions such as '0.15.2+224.geadfd92' instead. For an example of how to convert between, see python-versioneer/python-versioneer#45 (comment)

[shoyer]

I'll make a PR for this, which we can merge immediately before or after the release of v0.16.0. That way, dev versions will remain meaningfully ordered.

[kay1793]

if you actually read the PEP it makes it clear pandas itself is not allowed to use local version strings. just so you know.

[shoyer]

@kay1793 I did in fact read the PEP. My main intent here was to preserve the useful identifying nature of existing development versions. Git hashes are an important part of that.

We don't make or tag regular development releases of pandas (aside from release candidates), so the .dev and .post tags are not suitable for us. Because we don't distribute any releases with these versions, I think it's perfectly suitable to use local version identifiers in this context. If a user installs a development version of pandas, I think it is reasonable to consider that a "downstream" version of the upstream, released versions of pandas -- and in fact, this default format ensures that such downstream versions are sorted properly.

This usage is also exactly what versioneer settled on, and I think they provide a compelling justification in the PR adding this: python-versioneer/python-versioneer#67

[kay1793]

Because we don't distribute any releases with these versions, I think it's perfectly suitable to use
local version identifiers in this context

and the PEP says you're wrong.

If a user installs a development version of pandas, I think it is reasonable to consider that a
"downstream" version of the upstream, released versions of pandas

no.

We don't make or tag regular development releases of pandas (aside from release candidates),
so the .dev and .post tags are not suitable for us

"developmental release" is not clearly defined in the document, so .devN is probably what the PEP wants you to use. Otherwise, the PEP only concerns itself with official release versions and because pandas makes no official dev releases it can do what it likes during development and still be compliant (and users should use python setup.py).
In any case, choosing to treat every commit as if it were a developmental relase for versioning is far more reasonable then your claim that pandas is its own downstream (!) between releases.

Git hashes are an important part of that.

sure, but not for the setuptools version (all you need there is sorting). git hashes matter in the internal version pandas keeps in __version__, mainly for bug reports. that doesn't need to be pep440 compliant and is actually fine right now.

This usage is also exactly what versioneer settled on, and I think they provide a compelling
justification in the PR adding this: python-versioneer/python-versioneer#67

Do you mean:

So in one sense, we're abusing this feature to add information that doesn't fit in the 
other part (which PEP440 calls the "public version identifier"). But in another 
sense, tagged sources are "upstream", and your development tree (e.g. 
basically any source that isn't tagged) is a "downstream". 

?? that's not compelling at all. a project can't be its own downstream just like a person can't be his/her own mother.

[shoyer]

Do you really think that pandas should use different version identifiers in setup.py and __version__? That seems like a terrible idea to me.

Otherwise, the PEP only concerns itself with official release versions, and since pandasd makes no dev release, it can do what it likes during development and still be compliant

Yes, this is my point exactly. This is not a literal interpretation of PEP 440 guidelines but a pragmatic choice for a gray area that it doesn't cover (unreleased development versions).

The only reasonable alternative I've seen is '0.15.2.post.dev224+geadfd92', but that version string has a lot more noise.

Can you think of any actual situations where such usage would be problematic?

[shoyer]

You'll also notice:

Many build tools integrate with distributed version control systems like Git and Mercurial in order to add an identifying hash to the version identifier. As hashes cannot be ordered reliably such versions are not permitted in the public version field.

As with semantic versioning, the public .devN suffix may be used to uniquely identify such releases for publication, while the original DVCS based label can be stored in the project metadata.

Identifying hash information may also be included in local version labels. (emphasis mine)

https://www.python.org/dev/peps/pep-0440/#dvcs-based-version-labels

Pandas does not actually use a build system for unreleased development versions, so I'm not convinced it even makes sense to have .devN in the public version identifier, given that N here does not refer to any particular version (e.g., tracking master), but just however many commits have been made since the last tagged release (which may be squashed away or whatever).

[kay1793]

Do you really think that pandas should use different version identifiers in setup.py
and version? That seems like a terrible idea to me.

do you have a reason you'd like to share for thinking that?

I was thinking 0.15.2.dev242 and 0.15.2-242-geadfd92 are practically interchangable.

Yes, this is my point exactly. <...> this a pragmatic choice for a gray area that it doesn't cover ...

But it does define what an upstream project is and says you should not use local version strings. Whether you violate that in order to handle some other undefined need is beside the point.

Can you think of any actual situations where such usage would be problematic?

that's not how standards are supposed to work. violating them because "I don't see a problem" is not how standards should be treated. can you think of a reason to abuse a PEP when you don't need to?

[kay1793]

Identifying hash information may also be included in local version labels. (emphasis mine)

you're not allowed to use local version labels.

An "upstream project" is a project that defines its own public versions. A "downstream project" is one which tracks and redistributes an upstream project, potentially backporting security and bug fixes from later versions of the upstream project.

Local version identifiers SHOULD NOT be used for upstream projects.

Pandas does not actually use a build system for unreleased development versions

what does that have to do with anything?

 the public .devN suffix may be used to uniquely identify such releases

"uniquely identify" is what you care about, whether a "build system" is involved doesn't matter.

but just however many commits have been made since the last tagged release (which may
be squashed away or whatever).

so you're squshing public history in master now??

assuming you don't (you don't really) then the number of commits since the last tag is a unique (ordering) identifier of a commit and that's all you need for setuptools. The convenience of git tag can live in the (or in a) internal version string. setuptools doesn't need to know about it.

[shoyer]

so you're squshing public history in master now??

Assuming you don't (you don't really) then the number of commits since last tag can be used to uniquely identify a release, with sorting.

You can install development versions of pandas from source from any branch on any source, not just master. So IMO the number does not suffice.

Also, note that 0.15.2.devN < 0.15.2 as far as setup tools is concerned -- hence the need for .post.devN

[kay1793]

You can install development versions of pandas from source from any branch on any source,
not just master. So IMO the number does not suffice.

I don't understand. your goal is to control version sorting wrt to other commits in other forks and branches now? ok, well good luck with that.

Also, note that 0.15.2.devN < 0.15.2 as far as setup tools is concerned -- hence the need for .post.devN

Yes I did miss that and you actually need .post0.devN in that case I think.

Still waiting to hear why you think having pd.__version__ append "-ga1b3cdf" (or something) to the .devN version string in setup.py is such a "terrible idea". unless you actually meant that "all branches all sources" bit that is.

[ncoghlan]

I think using local versions for this is fine - the PEP 440 restriction is intended to be against uploading them to PyPI. Generating them when building directly from a source checkout seems fine to me - in that case, you're not relying on version specifiers, you're giving the VCS URL explicitly.

However, you may still want to consider incrementing the version number and switching to "commits since the last tag" dev release numbering - once the previous release has been published, installing from source control is at least arguably a pre-release of the next version, rather than a post-release of the previous one.

[kay1793]

@ncoghlan, The PEP (which you co-wrote... hi there) should say that then, because Local version identifiers SHOULD NOT be used for upstream projects. doesn't really. The PEP also strongly implies that local version strings are reserved for use by packagers and entities which distribute patched versions of an upstream package. The case of "version strings for installations from upstream git master" isn't covered which is why people are bending over backwards with silly arguments like "upstream is its own downstream" to work around what is written in there.

[tacaswell]

I think part of what is being lost here is that if a user is building from source then they are their own packagers. Using the down-stream post-fix (with the git hash) to make it easy for the upstream devs to understand exactly what version some one is using when they report an issue is a first order good (because 'master' is a moving target).

At my day-job we package and distribute internally a number of python packages (at this point mostly our own stuff) and I am sure that we are not unique in this regard. Having the upstream projects set up to emit sensible local patch numbering also seems like a first order good (because I am lazy).

@ncoghlan As I understood the pep the post can take a number ex X.X.X.postN+gHASH. If you do this with out a version bump you can get all of the information you need out of git describe rather than having to introspect the python (which is nice for playing nice with tools like conda).

There is also existential discussion of what a 'release' is. One can make the case that every git commit that passes tests is a 'release', but this seems like the wrong place to start painting that bike shed.