TST: warnings on the doc-tests

[jreback]

update at
#23149 (comment)

---

In the examples sections in the docstrings in pandas/core/generic.py we have two lines that when executed raise future warnings and should be changed to the latest syntax. The doctest validation does not report the exact lines (it reports line 1, which is wrong), but should be easy to find them by the error messages:

pandas/pandas/core/generic.py:1: FutureWarning: specifying 'categories' or 'ordered' in .astype() is deprecated; pass a CategoricalDtype instead
pandas/pandas/core/generic.py:1: FutureWarning: get_ftype_counts is deprecated and will be removed in a future version

The whole logs from the doctests are available in:

https://travis-ci.org/pandas-dev/pandas/jobs/441069332

Docs: https://docs.pytest.org/en/latest/warnings.html
============= 50 passed, 21 deselected, 1 warnings in 2.15 seconds =============
Doctests frame.py DONE
Doctests series.py
============================= test session starts ==============================
platform linux -- Python 3.6.6, pytest-3.8.2, py-1.6.0, pluggy-0.7.1 -- /home/travis/miniconda3/envs/pandas/bin/python
cachedir: .pytest_cache
rootdir: /home/travis/build/pandas-dev/pandas, inifile: setup.cfg
plugins: xdist-1.23.0, forked-0.2, cov-2.6.0, hypothesis-3.68.0
collecting ... collected 53 items / 4 deselected
pandas/core/series.py::pandas.core.series.Series.aggregate PASSED        [  2%]
pandas/core/series.py::pandas.core.series.Series.all PASSED              [  4%]
pandas/core/series.py::pandas.core.series.Series.any PASSED              [  6%]
pandas/core/series.py::pandas.core.series.Series.append PASSED           [  8%]
pandas/core/series.py::pandas.core.series.Series.apply PASSED            [ 10%]
pandas/core/series.py::pandas.core.series.Series.argmax PASSED           [ 12%]
pandas/core/series.py::pandas.core.series.Series.argmin PASSED           [ 14%]
pandas/core/series.py::pandas.core.series.Series.autocorr PASSED         [ 16%]
pandas/core/series.py::pandas.core.series.Series.between PASSED          [ 18%]
pandas/core/series.py::pandas.core.series.Series.combine PASSED          [ 20%]
pandas/core/series.py::pandas.core.series.Series.combine_first PASSED    [ 22%]
pandas/core/series.py::pandas.core.series.Series.corr PASSED             [ 24%]
pandas/core/series.py::pandas.core.series.Series.cummax PASSED           [ 26%]
pandas/core/series.py::pandas.core.series.Series.cummin PASSED           [ 28%]
pandas/core/series.py::pandas.core.series.Series.cumprod PASSED          [ 30%]
pandas/core/series.py::pandas.core.series.Series.cumsum PASSED           [ 32%]
pandas/core/series.py::pandas.core.series.Series.diff PASSED             [ 34%]
pandas/core/series.py::pandas.core.series.Series.drop PASSED             [ 36%]
pandas/core/series.py::pandas.core.series.Series.drop_duplicates PASSED  [ 38%]
pandas/core/series.py::pandas.core.series.Series.dropna PASSED           [ 40%]
pandas/core/series.py::pandas.core.series.Series.duplicated PASSED       [ 42%]
pandas/core/series.py::pandas.core.series.Series.fillna PASSED           [ 44%]
pandas/core/series.py::pandas.core.series.Series.idxmax PASSED           [ 46%]
pandas/core/series.py::pandas.core.series.Series.idxmin PASSED           [ 48%]
pandas/core/series.py::pandas.core.series.Series.isin PASSED             [ 51%]
pandas/core/series.py::pandas.core.series.Series.isna PASSED             [ 53%]
pandas/core/series.py::pandas.core.series.Series.isnull PASSED           [ 55%]
pandas/core/series.py::pandas.core.series.Series.map PASSED              [ 57%]
pandas/core/series.py::pandas.core.series.Series.memory_usage PASSED     [ 59%]
pandas/core/series.py::pandas.core.series.Series.nlargest PASSED         [ 61%]
pandas/core/series.py::pandas.core.series.Series.notna PASSED            [ 63%]
pandas/core/series.py::pandas.core.series.Series.notnull PASSED          [ 65%]
pandas/core/series.py::pandas.core.series.Series.nsmallest PASSED        [ 67%]
pandas/core/series.py::pandas.core.series.Series.prod PASSED             [ 69%]
pandas/core/series.py::pandas.core.series.Series.quantile PASSED         [ 71%]
pandas/core/series.py::pandas.core.series.Series.rename PASSED           [ 73%]
pandas/core/series.py::pandas.core.series.Series.replace PASSED          [ 75%]
pandas/core/series.py::pandas.core.series.Series.reset_index PASSED      [ 77%]
pandas/core/series.py::pandas.core.series.Series.sort_index PASSED       [ 79%]
pandas/core/series.py::pandas.core.series.Series.sort_values PASSED      [ 81%]
pandas/core/series.py::pandas.core.series.Series.sum PASSED              [ 83%]
pandas/core/series.py::pandas.core.series.Series.to_csv PASSED           [ 85%]
pandas/core/series.py::pandas.core.series.Series.to_excel PASSED         [ 87%]
pandas/core/series.py::pandas.core.series.Series.transform PASSED        [ 89%]
pandas/core/series.py::pandas.core.series.Series.unique PASSED           [ 91%]
pandas/core/series.py::pandas.core.series.Series.unstack PASSED          [ 93%]
pandas/core/series.py::pandas.core.series.Series.update PASSED           [ 95%]
pandas/core/series.py::pandas.core.series.Series.values PASSED           [ 97%]
pandas/core/series.py::pandas.core.series.Series.view PASSED             [100%]
========================== slowest 10 test durations ===========================
0.09s call     pandas/core/series.py::pandas.core.series.Series.replace
0.04s call     pandas/core/series.py::pandas.core.series.Series.fillna
0.04s call     pandas/core/series.py::pandas.core.series.Series.to_excel
0.03s call     pandas/core/series.py::pandas.core.series.Series.reset_index
0.02s call     pandas/core/series.py::pandas.core.series.Series.transform
0.02s call     pandas/core/series.py::pandas.core.series.Series.any
0.02s call     pandas/core/series.py::pandas.core.series.Series.notnull
0.02s call     pandas/core/series.py::pandas.core.series.Series.cummax
0.02s call     pandas/core/series.py::pandas.core.series.Series.cumprod
0.02s call     pandas/core/series.py::pandas.core.series.Series.cummin
=================== 49 passed, 4 deselected in 0.61 seconds ====================
Doctests series.py DONE
Doctests generic.py
============================= test session starts ==============================
platform linux -- Python 3.6.6, pytest-3.8.2, py-1.6.0, pluggy-0.7.1 -- /home/travis/miniconda3/envs/pandas/bin/python
cachedir: .pytest_cache
rootdir: /home/travis/build/pandas-dev/pandas, inifile: setup.cfg
plugins: xdist-1.23.0, forked-0.2, cov-2.6.0, hypothesis-3.68.0
collecting ... collected 69 items / 16 deselected
pandas/core/generic.py::pandas.core.generic.NDFrame.abs PASSED           [  1%]
pandas/core/generic.py::pandas.core.generic.NDFrame.add_prefix PASSED    [  3%]
pandas/core/generic.py::pandas.core.generic.NDFrame.add_suffix PASSED    [  5%]
pandas/core/generic.py::pandas.core.generic.NDFrame.asfreq PASSED        [  7%]
pandas/core/generic.py::pandas.core.generic.NDFrame.astype PASSED        [  9%]
pandas/core/generic.py::pandas.core.generic.NDFrame.at PASSED            [ 11%]
pandas/core/generic.py::pandas.core.generic.NDFrame.at_time PASSED       [ 13%]
pandas/core/generic.py::pandas.core.generic.NDFrame.between_time PASSED  [ 15%]
pandas/core/generic.py::pandas.core.generic.NDFrame.clip PASSED          [ 16%]
pandas/core/generic.py::pandas.core.generic.NDFrame.clip_lower PASSED    [ 18%]
pandas/core/generic.py::pandas.core.generic.NDFrame.clip_upper PASSED    [ 20%]
pandas/core/generic.py::pandas.core.generic.NDFrame.copy PASSED          [ 22%]
pandas/core/generic.py::pandas.core.generic.NDFrame.dtypes PASSED        [ 24%]
pandas/core/generic.py::pandas.core.generic.NDFrame.empty PASSED         [ 26%]
pandas/core/generic.py::pandas.core.generic.NDFrame.equals PASSED        [ 28%]
pandas/core/generic.py::pandas.core.generic.NDFrame.fillna PASSED        [ 30%]
pandas/core/generic.py::pandas.core.generic.NDFrame.filter PASSED        [ 32%]
pandas/core/generic.py::pandas.core.generic.NDFrame.first PASSED         [ 33%]
pandas/core/generic.py::pandas.core.generic.NDFrame.ftypes PASSED        [ 35%]
pandas/core/generic.py::pandas.core.generic.NDFrame.get_dtype_counts PASSED [ 37%]
pandas/core/generic.py::pandas.core.generic.NDFrame.get_ftype_counts PASSED [ 39%]
pandas/core/generic.py::pandas.core.generic.NDFrame.head PASSED          [ 41%]
pandas/core/generic.py::pandas.core.generic.NDFrame.iat PASSED           [ 43%]
pandas/core/generic.py::pandas.core.generic.NDFrame.iloc PASSED          [ 45%]
pandas/core/generic.py::pandas.core.generic.NDFrame.infer_objects PASSED [ 47%]
pandas/core/generic.py::pandas.core.generic.NDFrame.isna PASSED          [ 49%]
pandas/core/generic.py::pandas.core.generic.NDFrame.isnull PASSED        [ 50%]
pandas/core/generic.py::pandas.core.generic.NDFrame.last PASSED          [ 52%]
pandas/core/generic.py::pandas.core.generic.NDFrame.loc PASSED           [ 54%]
pandas/core/generic.py::pandas.core.generic.NDFrame.mask PASSED          [ 56%]
pandas/core/generic.py::pandas.core.generic.NDFrame.ndim PASSED          [ 58%]
pandas/core/generic.py::pandas.core.generic.NDFrame.notna PASSED         [ 60%]
pandas/core/generic.py::pandas.core.generic.NDFrame.notnull PASSED       [ 62%]
pandas/core/generic.py::pandas.core.generic.NDFrame.pop PASSED           [ 64%]
pandas/core/generic.py::pandas.core.generic.NDFrame.rename PASSED        [ 66%]
pandas/core/generic.py::pandas.core.generic.NDFrame.rename_axis PASSED   [ 67%]
pandas/core/generic.py::pandas.core.generic.NDFrame.replace PASSED       [ 69%]
pandas/core/generic.py::pandas.core.generic.NDFrame.sample PASSED        [ 71%]
pandas/core/generic.py::pandas.core.generic.NDFrame.set_axis PASSED      [ 73%]
pandas/core/generic.py::pandas.core.generic.NDFrame.size PASSED          [ 75%]
pandas/core/generic.py::pandas.core.generic.NDFrame.squeeze PASSED       [ 77%]
pandas/core/generic.py::pandas.core.generic.NDFrame.tail PASSED          [ 79%]
pandas/core/generic.py::pandas.core.generic.NDFrame.take PASSED          [ 81%]
pandas/core/generic.py::pandas.core.generic.NDFrame.to_clipboard PASSED  [ 83%]
pandas/core/generic.py::pandas.core.generic.NDFrame.to_csv PASSED        [ 84%]
pandas/core/generic.py::pandas.core.generic.NDFrame.to_hdf PASSED        [ 86%]
pandas/core/generic.py::pandas.core.generic.NDFrame.to_latex PASSED      [ 88%]
pandas/core/generic.py::pandas.core.generic.NDFrame.to_pickle PASSED     [ 90%]
pandas/core/generic.py::pandas.core.generic.NDFrame.to_sql PASSED        [ 92%]
pandas/core/generic.py::pandas.core.generic.NDFrame.to_xarray PASSED     [ 94%]
pandas/core/generic.py::pandas.core.generic.NDFrame.transform PASSED     [ 96%]
pandas/core/generic.py::pandas.core.generic.NDFrame.truncate PASSED      [ 98%]
pandas/core/generic.py::pandas.core.generic.NDFrame.where PASSED         [100%]
========================== slowest 10 test durations ===========================
0.21s call     pandas/core/generic.py::pandas.core.generic.NDFrame.to_xarray
0.19s call     pandas/core/generic.py::pandas.core.generic.NDFrame.to_sql
0.16s call     pandas/core/generic.py::pandas.core.generic.NDFrame.to_pickle
0.09s call     pandas/core/generic.py::pandas.core.generic.NDFrame.loc
0.08s call     pandas/core/generic.py::pandas.core.generic.NDFrame.truncate
0.07s call     pandas/core/generic.py::pandas.core.generic.NDFrame.replace
0.07s call     pandas/core/generic.py::pandas.core.generic.NDFrame.to_hdf
0.06s call     pandas/core/generic.py::pandas.core.generic.NDFrame.to_clipboard
0.05s call     pandas/core/generic.py::pandas.core.generic.NDFrame.iloc
0.04s call     pandas/core/generic.py::pandas.core.generic.NDFrame.fillna
=============================== warnings summary ===============================
/home/travis/build/pandas-dev/pandas/pandas/core/generic.py:1: FutureWarning: specifying 'categories' or 'ordered' in .astype() is deprecated; pass a CategoricalDtype instead
  # pylint: disable=W0231,E1101
/home/travis/build/pandas-dev/pandas/pandas/core/generic.py:1: FutureWarning: get_ftype_counts is deprecated and will be removed in a future version
  # pylint: disable=W0231,E1101
-- Docs: https://docs.pytest.org/en/latest/warnings.html
============= 53 passed, 16 deselected, 2 warnings in 1.71 seconds =============
Doctests generic.py DONE
Doctests top-level reshaping functions
============================= test session starts ==============================
platform linux -- Python 3.6.6, pytest-3.8.2, py-1.6.0, pluggy-0.7.1 -- /home/travis/miniconda3/envs/pandas/bin/python
cachedir: .pytest_cache
rootdir: /home/travis/build/pandas-dev/pandas, inifile: setup.cfg
plugins: xdist-1.23.0, forked-0.2, cov-2.6.0, hypothesis-3.68.0
collecting ... collected 8 items / 4 deselected
pandas/core/reshape/concat.py::pandas.core.reshape.concat.concat PASSED  [ 25%]
pandas/core/reshape/pivot.py::pandas.core.reshape.pivot.pivot PASSED     [ 50%]
pandas/core/reshape/reshape.py::pandas.core.reshape.reshape._Unstacker PASSED [ 75%]
pandas/core/reshape/reshape.py::pandas.core.reshape.reshape.get_dummies PASSED [100%]
========================== slowest 10 test durations ===========================
0.06s call     pandas/core/reshape/concat.py::pandas.core.reshape.concat.concat
0.05s call     pandas/core/reshape/pivot.py::pandas.core.reshape.pivot.pivot
0.05s call     pandas/core/reshape/reshape.py::pandas.core.reshape.reshape.get_dummies
0.01s call     pandas/core/reshape/reshape.py::pandas.core.reshape.reshape._Unstacker
0.00s setup    pandas/core/reshape/concat.py::pandas.core.reshape.concat.concat
0.00s setup    pandas/core/reshape/reshape.py::pandas.core.reshape.reshape._Unstacker
0.00s setup    pandas/core/reshape/pivot.py::pandas.core.reshape.pivot.pivot
0.00s setup    pandas/core/reshape/reshape.py::pandas.core.reshape.reshape.get_dummies
0.00s teardown pandas/core/reshape/reshape.py::pandas.core.reshape.reshape.get_dummies
0.00s teardown pandas/core/reshape/concat.py::pandas.core.reshape.concat.concat
=============================== warnings summary ===============================
/home/travis/build/pandas-dev/pandas/pandas/core/reshape/concat.py:1: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version
of pandas will change to not sort by default.
To accept the future behavior, pass 'sort=False'.
To retain the current behavior and silence the warning, pass 'sort=True'.
  """
-- Docs: https://docs.pytest.org/en/latest/warnings.html
============== 4 passed, 4 deselected, 1 warnings in 0.21 seconds ==============
Doctests top-level reshaping functions DONE

[jreback]

couple of instances of warnings from the doc-tests themselves.

cc @datapythonista @jorisvandenbossche @TomAugspurger

[datapythonista]

Didn't see those. Updated the description to make it easier for first time contributors to pick.

[jorisvandenbossche]

I am not fully sure what we can do about this in general.
Of course the ones that are using deprecated keywords can be updated (like the astype one). But the other is simply a deprecated attribute. So the only way to get rid of the warnings, would be to remove the documentation altogether, something we don't want to do I think.

Still, it would be nice to better catch this. One option might be to skip those docstrings of deprecated attributes/methods (with # doctest: +SKIP), so that we can actually error on the ones that actually should be solved?

[TomAugspurger]

From https://api.travis-ci.org/v3/job/451403135/log.txt we're down to two warnings I think

�[33m=============================== warnings summary ===============================�[0m
pandas/core/frame.py::pandas.core.frame.DataFrame.to_parquet
  /home/travis/build/pandas-dev/pandas/pandas/io/parquet.py:123: ResourceWarning: unclosed file <_io.BufferedReader name='df.parquet.gzip'>
    **kwargs).to_pandas()

�[33m=============================== warnings summary ===============================�[0m
pandas/core/generic.py::pandas.core.generic.NDFrame.get_ftype_counts
  /home/travis/build/pandas-dev/pandas/pandas/core/generic.py:1: FutureWarning: get_ftype_counts is deprecated and will be removed in a future version
    # pylint: disable=W0231,E1101

I can't reproduce the parquet one.

I'd recommend just skipping the get_ftype_counts one.

pytest --doctest-modules pandas/core/generic.py -k get_ftype_counts

[datapythonista]

I think the get_ftype_counts should be gone with #23552

I didn't see the parquet one locally, but I think we'll be skipping that test in #23201 anyway.

[TomAugspurger]

Works for me. Thanks or checking.