ENH: handle multiple tbody in read_html()

[jstray]

Code Sample

url = 'https://www.wunderground.com/history/airport/KORD/2018/3/21/CustomHistory.html?dayend=10&monthend=4&yearend=2018'
df = pandas.read_html(url)[1]

Problem description

Expected: table of weather information at page bottom.

Actual:
0.21.1 - first two rows
0.22.0 - exception
current master - first two rows

For reference, Google Sheets IMPORTHTML loads this table correctly.,

[WillAyd]

It looks like the second table has multiple tbody tags but the parser only looks at the first:

pandas/pandas/io/html.py

Line 394 in ad5affb

res = self._parse_tr(tbody[0])

PRs welcome

[jstray]

Thanks. I'll probably have to fix this.

[gfyoung]

We could always enhance read_html to behave like read_excel where we can read in multiple tbody elements. That doesn't seem too unreasonable for the moment.

[jstray]

How is read_excel similar to this? There’s no tbody in that case.

[WillAyd]

@gfyoung are you just referring to the ability to return multiple tables? read_html technically already does that as it returns a list instead of just a DataFrame, but chime in if I misunderstand.

@jstray one design consideration to think about - should multiple tbody elements return a MultiIndexed DataFrame? I suppose having those multiple tbody tags in the first place is indicative of different groupings within the table, so maybe it makes sense for each of them to be a unique value within the first level of a MultiIndex?

Not saying that needs to happen at the outset as certainly being able to just parse them would be an improvement over what we have today. Just throwing that out there as food for thought as you try a patch

[gfyoung]

@WillAyd : Sorry, misspoke there. Please ignore that comment. 😄