API Docs Rework

[TomAugspurger]

What do we want out of our API docs? What are the current issues?

I think we're (mostly) happy with api.rst: we manually list the classes and methods that we want to include the docs, grouped by topics of our choosing.

I think we're unhappy with some aspects of the auto-genrated class pages.

The hack in our vendored numpydoc.py "fixing" the autosummary to not include a table of methods for certain classes. This avoids a ton of redundant pages with the parent class, but we can't call out important differences like RangeIndex.start

We'd like a way to limit autosummary to specific methods in a class (I think this can be done with a Methods section in the docstring)

Tasks:

• Limit API doc of classes to certain members (either extend autodoc or do it manually?) -> DOC: document subclasses in API docs with selection of specific methods/attributes #18202
• DOC: fix API docs generation with sphinx 1.6 DOC: fix API docs generation with sphinx 1.6 #16705
• DOC: documenting cython class (eg Timestamp): "cyfunction is not a python function" DOC: documenting cython class (eg Timestamp): "cyfunction is not a python function" #5218 -> DOC: document subclasses in API docs with selection of specific methods/attributes #18202
• Remove vendored IPython sphinxext (DOC: Remove vendored IPython.sphinext #18193, DOC: removed vendored IPython.sphinxext - take 2 #19657)
  • should be OK with BUG sphinxext: ensure sys.stdout is used for logging (#10904) ipython/ipython#10907 released)
  • removed in DOC: removed vendored IPython.sphinxext - take 2 #19657
  • We should check on Windows!
• Align our numpydoc version with upstream / use upstream version: for this we somehow need to fix (or push upstream) our hacks to numpydoc ( I think this are the only relevant changes to the numpydoc source since it was last aligned with upstream in DOC: update numpydoc to current master (commit 223df02530) #6003) (xref DOC: some hacks to get rid of warnings #11257)):
  • cf40991 initial commit for allowing classes without members table -> DOC: document subclasses in API docs with selection of specific methods/attributes #18202
  • 70c9d31 for None attributes
  • Remove vendored numpydoc (work around Format attributes like parameters numpy/numpydoc#106 (comment))
• Fix cache_readonly copying docstring (The use of @cache_readonly removes the docstring of attributes #18197)
• Index.name and other class attributes. Either fix in numpydoc to use sphinx magic, or make into properties
• DataFrame.index and DataFrame.columns (Allow accessing AxisProperties on classes #18196)
• Check performance (better than 1.6, but still slower? Considerable slowdown in writing autodoc pages with sphinx 1.6.x sphinx-doc/sphinx#4230)

[jorisvandenbossche]

I indeed think we at least want to have a way to only document certain methods/attributes on a class, to prevent including all.

Something like outlined above (with the :members: to the autosummary directive) is nice. But if we don't directly find a solution, it might be easier to go this way: for those 'problematic' classes, include the actual docstring page (which is now autogenerated, but for some classes it would not be that much work to make them ourselves) and then you can explicitly include those members me want.

[jorisvandenbossche]

I think the above is somewhat the minimal we need to be able to document all classes, without an explosion of docstring pages.
But, there are also other features which could be nice. Eg more automate this, and if docstrings are identical, only build it once for the base Index class, and have it for other classes link to that single docstring page

We could also try to limit the pages we currently already have (eg remove the duplicate pages for DatetimeIndex and TimedeltaIndex), but then we might need to provide redirects.

[jorisvandenbossche]

So some other related issue to the API docs:

• DOC: fix API docs generation with sphinx 1.6 DOC: fix API docs generation with sphinx 1.6 #16705
• DOC: documenting cython class (eg Timestamp): "cyfunction is not a python function" DOC: documenting cython class (eg Timestamp): "cyfunction is not a python function" #5218
• Align our numpydoc version with upstream / use upstream version: for this we somehow need to fix (or push upstream) our hacks to numpydoc (cf40991 initial commit for allowing classes without members table and 70c9d31 for None attributes -> I think this are the only relevant changes to the numpydoc source since it was last aligned with upstream in DOC: update numpydoc to current master (commit 223df02530) #6003) (xref DOC: some hacks to get rid of warnings #11257)

[TomAugspurger]

For moving to IPython.sphinxext, I think the only new warnings are

source/whatsnew/v0.7.3.txt:None: WARNING: image file not readable: _static/scatter_matrix_kde.png
source/whatsnew/v0.7.3.txt:None: WARNING: image file not readable: _static/bar_plot_stacked_ex.png
source/whatsnew/v0.7.3.txt:None: WARNING: image file not readable: _static/barh_plot_stacked_ex.png

which is good because those figures don't exist. They're in source/savefig. Oh, I see that's a change from prior versions, so it used to be correct. It should be controllable with ipython_savefig_dir. https://ipython.org/ipython-doc/3/api/generated/IPython.sphinxext.ipython_directive.html#module-IPython.sphinxext.ipython_directive

[jorisvandenbossche]

If we want to go completely warning free, we also have those ones in the beginning of the build:

WARNING: [autosummary] failed to import 'pandas.DataFrame.columns': no module named pandas.DataFrame.columns
WARNING: [autosummary] failed to import 'pandas.DataFrame.index': no module named pandas.DataFrame.index
WARNING: [autosummary] failed to import 'pandas.Panel.items': no module named pandas.Panel.items
WARNING: [autosummary] failed to import 'pandas.Panel.major_axis': no module named pandas.Panel.major_axis
WARNING: [autosummary] failed to import 'pandas.Panel.minor_axis': no module named pandas.Panel.minor_axis
WARNING: [autosummary] failed to import 'pandas.Series.index': no module named pandas.Series.index

[TomAugspurger]

@jorisvandenbossche not sure if this will work for the cyfuncs, but see the second example here

Perhaps if we put the function signature on the first line of the docstring, numpydoc / sphinx will not attempt to introspect the cyfunc?

[jorisvandenbossche]

Perhaps if we put the function signature on the first line of the docstring, numpydoc / sphinx will not attempt to introspect the cyfunc?

That might be an option (at least worth a try), but, that means we need to override all those inherited methods I think ?

For the attributes listing, I commented here: numpy/numpydoc#106 (comment)

[jorisvandenbossche]

Some progress: IPython fix is merged: ipython/ipython#10907, and I also opened a PR to sphinx (for the missing listing of attributes): sphinx-doc/sphinx#4247
But for sphinx 1.6, the main bottleneck is still the slowdown (sphinx-doc/sphinx#4230)