REGR: pd.to_hdf(dropna=True) not dropping all nan rows

[XiaozhanYang]

Location of the documentation

[this should provide the location of the documentation, e.g. "pandas.read_csv" or the URL of the documentation, e.g. "https://dev.pandas.io/docs/reference/api/pandas.read_csv.html"]

Note: You can check the latest versions of the docs on master here.

https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html

Documentation problem

[this should provide a description of what documentation you believe needs to be fixed/improved]

On the section:

"HDFStore will by default not drop rows that are all missing. This behavior can be changed by setting dropna=True."

The last example:

In [370]: pd.read_hdf('file.h5', 'df_with_missing')
Out[370]:
col1 col2
0 0.0 1.0
1 NaN NaN
2 2.0 NaN

should output:

col1 col2
0 0.0 1.0
1 2.0 NaN

which does not have rows that are all missing.

Suggested fix for documentation

[this should explain the suggested fix and why it's better than the existing documentation]

This has been provided above.

[simonjayhawkins]

Thanks @XiaozhanYang for the report.

The docs are built from the code samples and so this is a bug and not directly a documentation issue. #35720 (comment)

This is a regression between 0.25.3 and 1.0.5 and persists on master

>>> pd.__version__
'0.25.3'
>>>
>>> df_with_missing = pd.DataFrame({"col1": [0, np.nan, 2], "col2": [1, np.nan, np.nan]})
>>> df_with_missing
   col1  col2
0   0.0   1.0
1   NaN   NaN
2   2.0   NaN
>>>
>>> df_with_missing.to_hdf(
...     "file.h5", "df_with_missing", format="table", mode="w", dropna=True
... )
>>> result = pd.read_hdf("file.h5", "df_with_missing")
>>> result
   col1  col2
0   0.0   1.0
2   2.0   NaN
>>>

>>> pd.__version__
'1.0.5'
>>>
>>> df_with_missing = pd.DataFrame({"col1": [0, np.nan, 2], "col2": [1, np.nan, np.nan]})
>>> df_with_missing
   col1  col2
0   0.0   1.0
1   NaN   NaN
2   2.0   NaN
>>>
>>> df_with_missing.to_hdf(
...     "file.h5", "df_with_missing", format="table", mode="w", dropna=True
... )
>>> result = pd.read_hdf("file.h5", "df_with_missing")
>>> result
   col1  col2
0   0.0   1.0
1   NaN   NaN
2   2.0   NaN
>>>