Generate Misc/NEWS from individual files

[brettcannon]

As pulled from PEP 512:

Traditionally the Misc/NEWS file [19] has been problematic for changes which spanned Python releases. Oftentimes there will be merge conflicts when committing a change between e.g., 3.5 and 3.6 only in the Misc/NEWS file. It's so common, in fact, that the example instructions in the devguide explicitly mention how to resolve conflicts in the Misc/NEWS file [21] . As part of our tool modernization, working with the Misc/NEWS file will be simplified.

The planned approach is to use an individual file per news entry, containing the text for the entry. In this scenario each feature release would have its own directory for news entries and a separate file would be created in that directory that was either named after the issue it closed or a timestamp value (which prevents collisions). Merges across branches would have no issue as the news entry file would still be uniquely named and in the directory of the latest version that contained the fix. A script would collect all news entry files no matter what directory they reside in and create an appropriate news file (the release directory can be ignored as the mere fact that the file exists is enough to represent that the entry belongs to the release). Classification can either be done by keyword in the new entry file itself or by using subdirectories representing each news entry classification in each release directory (or classification of news entries could be dropped since critical information is captured by the "What's New" documents which are organized). The benefit of this approach is that it keeps the changes with the code that was actually changed. It also ties the message to being part of the commit which introduced the change. For a commit made through the CLI, a script could be provided to help generate the file. In a bot-driven scenario, the merge bot could have a way to specify a specific news entry and create the file as part of its flattened commit (while most likely also supporting using the first line of the commit message if no specific news entry was specified). If a web-based workflow is used then a status check could be used to verify that a new entry file is in the pull request to act as a reminder that the file is missing. Code for this approach has been written previously for the Mercurial workflow at http://bugs.python.org/issue18967 . There is also tools from the community like https://pypi.python.org/pypi/towncrier , https://github.com/twisted/newsbuilder , and http://docs.openstack.org/developer/reno/ .

Discussions at the Sep 2016 Python core-dev sprints led to this decision compared to the rejected approaches outlined in the Rejected Ideas section of this PEP. The separate files approach seems to have the right balance of flexibility and potential tooling out of the various options while solving the motivating problem.

[brettcannon]

Rejected Ideas

Deriving Misc/NEWS from the commit logs

As part of the discussion surrounding Handling Misc/NEWS , the suggestion has come up of deriving the file from the commit logs itself. In this scenario, the first line of a commit message would be taken to represent the news entry for the change. Some heuristic to tie in whether a change warranted a news entry would be used, e.g., whether an issue number is listed.

This idea has been rejected due to some core developers preferring to write a news entry separate from the commit message. The argument is the first line of a commit message compared to that of a news entry have different requirements in terms of brevity, what should be said, etc.

Deriving Misc/NEWS from bugs.python.org

A rejected solution to the NEWS file problem was to specify the entry on bugs.python.org [5] . This would mean an issue that is marked as "resolved" could not be closed until a news entry is added in the "news" field in the issue tracker. The benefit of tying the news entry to the issue is it makes sure that all changes worthy of a news entry have an accompanying issue. It also makes classifying a news entry automatic thanks to the Component field of the issue. The Versions field of the issue also ties the news entry to which Python releases were affected. A script would be written to query bugs.python.org for relevant new entries for a release and to produce the output needed to be checked into the code repository. This approach is agnostic to whether a commit was done by CLI or bot. A drawback is that there's a disconnect between the actual commit that made the change and the news entry by having them live in separate places (in this case, GitHub and bugs.python.org). This would mean making a commit would then require remembering to go back to bugs.python.org to add the news entry.

[brettcannon]

The only other potential solution other than individual files is a bot or script which collects news entry from messages in PR comments. The pro/con list is:

Individual files

Pro

• Keeps NEWS entry with commit
• All PR submitters learn about the NEWS file and thus a little bit more about our process

Con

• Requires the PR submitter to create the NEWS entry which makes contributions slower

Bot

Pros

• Don't need to lean on PR submitters to create the NEWS entry

Cons

• Another bot to maintain would be annoying
• Script would need a way to keep track of what entries have been added to the NEWS file thus far so that there wasn't just a constant growth of the number of PRs that had to be scraped

[dhellmann]

Managing individual files by hand also has the benefit of having the ability to communicate separately to reviewers via commit messages and consumers via the release notes. We've had good luck with that in OpenStack where the consumer of the software is often interested in very different information from the other contributors who may be reviewing patches.

[dhellmann]

Hmm, it looks like the sample repo in python/cpython doesn't have any tags indicating when specific versions were released. Reno currently relies on tags for version identification. I could update it to look at something else -- how does one determine the current point release on a given branch? Is it in a file somewhere?

[orsenthil]

@dhellmann , yeah, that's a problem with the current python/cpython repo. Tags were not replicated and pushed. We will have to repush the repo with tags (perhaps using tool and/or instructions from https://github.com/orsenthil/cpython-hg-to-git) which has been tried.

[dhellmann]

@orsenthil oh, good, if the plan is to eventually have the tags in place then that's no problem. I assumed there was some other mechanism in place already.

[dstufft]

This tool should probably make this pretty easy https://pypi.org/project/towncrier/

[vstinner]

I started a thread on python-dev because Misc/NEWS became a blocker point with the new GitHub workflow:
https://mail.python.org/pipermail/python-dev/2017-February/147417.html

---

FYI I wrote a tool which computes the released Python versions including a change from a list of commits to generate a report on Python vulnerabilities:
https://github.com/haypo/python-security/blob/master/render_doc.py

The core of the feature is "git tag --contains sha1". Then I implemented a logic to select which versions should be displayed. Output:
http://python-security.readthedocs.io/vulnerabilities.html

The tool also computes the number of days between the vulnerability disclosure date, commit date and release date. I chose to ignore beta and release candidate versions.

But I guess that there are other existing projects which would fit Misc/NEWS requirements better than my tool! (reno, twoncrier, something else?)

[westurner]

changelog filenames

• CHANGELOG.rst
• HISTORY.rst
• whatsnew.rst
• Misc/NEWS

Escaping Markup

NOTE: commit logs may contain (executable) markup

• https://wiki.python.org/moin/EscapingHtml
• https://github.com/westurner/dotfiles/blob/develop/scripts/git-changelog.py
  • rst_escape() (!?)

Tools

• mergenews

  • Src: https://bitbucket.org/larry/mergenews
  • splitnews
  • mergenews
  • pyci
    • hg-specific

• Reno

  • Src: https://github.com/openstack/reno
  • Src: https://github.com/openstack/reno/blob/master/reno/formatter.py
  • Src: https://github.com/openstack/reno/blob/master/reno/sphinxext.py
  • PyPI: https://pypi.org/project/reno
  • Docs: https://docs.openstack.org/developer/reno/usage.html#editing-a-release-note
  • ENH: "make sections configurable" openstack/reno@081a414

• Towncrier

  • Src: https://github.com/hawkowl/towncrier
  • PyPI: https://pypi.org/project/towncrier

Projects:

• CPython
  • https://github.com/python/cpython/blob/master/Misc/NEWS
  • https://github.com/python/cpython/tree/master/Doc/whatsnew
  • https://docs.python.org/3/whatsnew/
  • https://docs.python.org/3/whatsnew/3.6.html
• IPython
  • https://github.com/ipython/ipython/tree/master/docs/source/whatsnew/pr#documenting-whats-new
  • https://github.com/ipython/ipython/tree/master/docs/source/whatsnew
  • Src: https://github.com/ipython/ipython/blob/master/tools/update_whatsnew.py
• Matplotlib
  • https://github.com/matplotlib/matplotlib/blob/master/doc/users/whats_new.rst
  • https://github.com/matplotlib/matplotlib/tree/master/doc/users/prev_whats_new
  • https://github.com/matplotlib/matplotlib/tree/master/doc/users/whats_new README.rst
• scikit-learn
  • ENH, BUG, DOC, MRG
  • https://github.com/scikit-learn/scikit-learn/blob/master/doc/whats_new.rst
• statsmodels
  • ENH, BUG, DOC, MRG
  • https://github.com/statsmodels/statsmodels/tree/master/docs/source/release
• OpenStack Nova
  • reno
  • https://git.openstack.org/cgit/openstack/nova/tree/releasenotes/notes
  • https://github.com/openstack/nova/tree/master/releasenotes/notes
  • http://lists.openstack.org/pipermail/release-announce/2017-February/000784.html

[westurner]

Additional requirements/requests for Misc/NEWS'ification

• SEC: Security
  • indicate that a releasenotedchange is security-relevant
  • link w/ @Haypo render_doc.py
    • reno YAML would be flexible here

[brettcannon]

Since there seem to be multiple options for pre-existing tools and writing our own, we should start by identifying the requirements of what we want to be supported in the final NEWS file:

Features we want

• Sectioned by Python release with the release date
• Release sub-sectioned by topic
• Issues referenced along with explanation of what changed
• Simple bullet list
• Single file

Nice-to-haves would be:

• Reference changed/affected module(s) for per-module grouping (and a "general" for multi-module changes)

Am I missing anything we need/want the solution to cover?

What if we were writing a tool from scratch?

Now what would a greenfield solution look like (to help set a baseline of what we might want a tool to do)? To me, we would have a directory to contain all news-related details. In that top-level directory would be subdirectories for each subsection of the release (e.g. "Core and Built-ins", "Library", etc.). Each NEWS entry file would then go into the appropriate subdirectory. The filename would be the issue related to the change, e.g. bpo-12345.rst (in all honesty we should have an issue per change in the NEWS file as a requirement since if it's important enough to be listed in the NEWS files then we need a way to track any past and future discussions related to the change, and yes, we can come up with a standard for listing multiple issue numbers). If we wanted to support listing affected module(s) we could have some convention in the files to list that sort of detail (e.g. "modules: importlib").

Then upon release the RM would run a script which would read all the files, generate the appropriate text (which includes line-wrapping, etc.), and write out the file. Now we can either keep a single file with all entries (which gets expensive to load and view online, which also means we should add an appropriate .rst file extension), or a file per feature release (which makes searching the complete history a little harder as you then have to use something like grep to search multiple files quickly). I'm assuming we won't feel the need to drag NEWS entries forward -- e.g. entries made for 3.6.1 won't be pulled into the master branch for 3.7 -- since most changes in bugfix branches will also be fixed in the master branch and thus be redundant. But if people really care then a single PR to copy the update NEWS file(s) to master upon release could be done.

Next steps

First step is to figure out what our requirements are of any solution. So if I'm missing anything please reply here (I think I covered what @warsaw mentioned on python-dev above). Once we have the requirements known we can look at the available tools to see how close they come to meeting our needs and decide if we like any of them or want to write our own.