Deprecation of Panel ?

[jorisvandenbossche]

This is a topic that has come up recently (#10000, #8906, pandas-dev mailing list discussion), let's make this an issue to track the discussion about it.

• Partial indexing of a Panel #8906 (comment) experience of the 'experimental' status of Panels by @MaximilianR and following discussion on pointing users to xarray
• Issue for docs on using xarray instead of Panels: DOC: show how to use to_xarray then use xarray to perform analysis on Panels #12283
• "WIP for transitioning from Panel docs" PR WIP for transitioning from Panel docs pydata/xarray#832

Deprecating Panels would be a rather large change, so:

• Do we need to further discuss if we actually want to do this?
• Are there people who make intensive use of Panels to ask feedback?
• How do we go about such a deprecation? First making a note in the whatsnew / pinging mailing list or other fora before actually deprecating?

cc @pydata/pandas @MaximilianR

[sinhrks]

I'm +1 on moving to xarray, but GitHub search shows the deprecation is not easy... As long as I know about popular packages, pydata/data-reader and quantopian/zipline uses Panel.

CC @davidastephens @ehebert

[max-sixty]

No change this end - we are still using xarray heavily, and it's working beautifully. We've also improved the integration of xarray & pandas, so that should ease the path to deprecation.

[wesm]

I'm +1 on deprecating Panels; @jreback moved mountains to create a consistent internal object model from 1 to N dimensions, but there is still a feeling of second-class citizenry when it comes to working with data over 2 dimensions. I think we would be better served in the long run by really optimizing for the 1 and 2-dimensional use cases (similar to what the R community has done, though the API surface area of dplyr, data.table, and built-in data frames is quite a bit smaller than pandas -- primarily lacking in the level of indexing complexity).

I maintain that we should plan for a pandas 0.X.Y long-term support LTS release branch that becomes bugfix only so that we can start investing in renovations. I'm interested in feedback from the other core devs how realistic you feel this is.

I've long worried about the amount of baggage we are carrying forward -- there are many organizations with large codebases that have made their peace with pandas's rough edges (data type issues, view / copying semantics, etc.), and it doesn't make sense to abandon them. On the flip side, it would be a shame to be held back from undertaking a more aggressive cleanup and retool of the internals to introduce better performance, extensibility, missing data / data type issues, etc. I regret that 6 months have passed since I brought up this grand scheme and I haven't been able to carve out the time to make a dent, beyond demo'ing a proof-of-concept of integer NAs. Also, I would feel much better about working on this on a long-lived branch (similar to what happened with IPython) under some kind of feature freeze.

Anyway, some of these comments are beyond the scope of this issue. I don't think we should deprecate Panel unless we're collectively on board to the idea of cleaning up pandas internals over the next 12-24 months (which is as much of a code organization problem as anything -- particularly quarantining unit tests that we are contemplating "breaking").