DOC: use Sphinx extension directly from numpy/ipython instead of maintaining our own version

[jorisvandenbossche]

Summary: proposal to not maintain our own version of Sphinx extension in pandas repo but instead use the extensions directly from the project it maintains.

The pandas docs are using some Sphinx extensions that are not included in Sphinx itself: numpydoc and ipythons ipython_directive and ipython_console_highlighting (also matplotlibs plot_directive is imported in conf.py, but this is not included in the pandas repo).
These extension are located here: https://github.com/pydata/pandas/tree/master/doc/sphinxext.

I think that ideally pandas does not have to care about them, but just uses them by importing the extension from the respective project it lives under (numpy/ipython).
Some recent issue with the doc building also arised form outdated versions of these extensions in pandas repo (see part of the commits in PR #5160 and #4165).

The original extensions are living here:

• ipython sphinxext in ipython itself: https://github.com/ipython/ipython/tree/master/IPython/sphinxext
• numpydoc in its own repo: https://github.com/numpy/numpydoc

Options we could do:

• ipython extensions can just be imported from there in conf.py (because ipython is already a dependency)
• numpydoc could be added as a submodule (this is how eg numpy itself does it: https://github.com/numpy/numpy/tree/master/doc, 'sphinxext' links to numpydoc), or it can be added as a doc build dependency.

Possible problems with this is that the version in the pandas repo has converged from the version in numpy/ipython:

• I think for numpydoc this will not be a problem. Following the history it was updated 3 years ago, and then only recently for python 3 compatibility.
• For ipython_directive this seems more of a problem, looking at the history: https://github.com/pydata/pandas/commits/master/doc/sphinxext/ipython_directive.py there seems to be work done on this specifically for the pandas docs. If this is the case that is has converged from the ipython one, then ideally this could be contributed back to ipython (if the feature is not available there), but this will be more work to adapt.

[jreback]

@jorisvandenbossche this all sounds great.....+1 to decrease these types of deps

[jtratner]

I'm fine with this change and if we can rely on ipython and numpy that sounds good to me. The only problem that I see is that this potentially ties us to specific numpy and IPython versions. When you're changing this, can you make sure that the builds pass (i.e., at least build even if they generate warnings/failures because of numpy compat) with numpy 1.6, numpy 1.7, the numpy 1.8rc and numpy master, plus IPython 1.0 and IPython master. (maybe IPython 0.9?). We don't have to add them to Travis, but I want to make sure that we're not entering some new issue with doc building.

[jorisvandenbossche]

I think this can be an advantage of adding it as a submodule (although I don't really have experience with it), as you can include it at a certain commit (and then periodically update it).
Eg for numpydoc it could be done like this. Numpydoc is also splitted into its own repo, so you wouldn't depend on the numpy version.

For ipython it could indeed be more difficult with different version of ipython itself, because it is included in ipython core.

And for clarity, at the moment I just posted it here as an idea to discuss :-) (I have at the moment unfortunately not really the bandwith to look at it)

[jtratner]

I'd rather have it be an external dep or have it checked into pandas proper. Submodules can be obnoxious.

[jseabold]

To add some historical perspective. The separate numpydoc is a recent venture, and the numpydoc extension couldn't always be depended on to be available just because numpy was installed. Similarly ipython_directive in IPython was broken for a long time as it wasn't much used and didn't keep up with internal IPython refactorings. It was fixed (and somewhat maintained in statsmodels) outside of IPython for a while, but not so much anymore.

Long story short, the status quo has changed, so I'm +1 on going back to using the 'official' extensions.

[jorisvandenbossche]

@jseabold Thanks for the explanation!

Do you know anything about the work done on the ipyton directive in pandas? (see the history: https://github.com/pydata/pandas/commits/master/doc/sphinxext/ipython_directive.py, there seems to be some changes done to it)

[jseabold]

No, not since Wes' last changes. If there are issues they should be sent to upstream IPython I think. That's what I'd been doing.

[jorisvandenbossche]

@jseabold The more recent changes are only minor or not relevant I think (mainly py3 compat => ipython now is also compatible, and something copied over from the ipython version), wo maybe they wouldn't cause to much problems when converting to the 'official' extension.
But most of the changes are from the last change of Wes and before (mainly work on the pure python code blocks it seems). Do you know about that? Are they in upstream?

[jseabold]

I couldn't say for sure without looking. These are the only places this would've changed though other than pandas.

https://github.com/ipython/ipython/blob/master/IPython/sphinxext/ipython_directive.py
https://github.com/statsmodels/statsmodels/blob/master/docs/sphinxext/ipython_directive.py

[cpcloud]

I wrote a hack to allow Cython to work, I think there's one other thing, but I can't remember exactly what it is. I could potentially submit a PR to IPython to put the Cython hack in ...

[jtratner]

Also changed the setting for the IPython cache too right?