DEPR: SparseDataFrame and SparseSeries subclasses

[jreback]

This is slightly useful in a single dtyped case, but enough of a headache here that we should simply remove it. Ultimately we should even remove SparseSeries and just rely on the SparseArray abstraction and extension types (#19174) .

[jreback]

cc @jorisvandenbossche @TomAugspurger @wesm
cc @hexgnu
cc @Licht-T
cc @kernc

[datapythonista]

In case my use case it's useful in the discussion. I mostly (probably only) use SparseDataFrame because of pandas.get_dummies(). And after that, I'm mainly interested in having a scipy.sparse structure (for example to use it in scikit-learn). I'd bet this is the case for many pandas users.

Personally I'm +1 on getting rid of it. Then, it could be useful to have a get_dummies sparse version that returns a scipy.sparse. Not sure if in pandas or as a separate project.

[hexgnu]

I think a lot of the issue with SparseDataFrame is honestly a lack of test coverage as well as a lot of bleedover into DataFrame. For instance a lot of code assumes that something is dense and that it has length > 0. This isn't the real case, and I feel that with a flag day or two of squashing sparse bugs and writing more tests it wouldn't be so awful.

But that being said I also don't want to spend a bunch of time fixing something nobody really uses.

Also this is not obvious to me on first look:

• DataFrame can have both Series and SparseSeries in it
• SparseDataFrame can only really have SparseSeries inside of it

Personally I would think that SparseDataFrame is the only time you can have SparseSeries and DataFrame is always dense.

[hexgnu]

Also I wanted to point out that Sparse bugs aren't the biggest label category in this repo

It looks to me like reshaping, indexing, and dtypes are the big hotspots. Getting rid of sparse I don't think would bring those bug counts down.

Also I think with some TLC SparseDataFrames can be a really nice little abstraction for saving on memory consumption in certain cases.

[jreback]

@hexgnu its not about the issues, rather about having another structure to use, which leads to confusion and code complexity. Yes a pure sparse frame does lead to efficiencies, but you (w/o a lot of acrobatics) lose the heterogenity which makes a DataFrame itself so useful. I think a SparseDataFrame should exist as a separate project :> (and the primitives, IOW SparseArray/Series can still live in pandas proper).

[kernc]

As long as high-level NDFrame APIs defer actions to block primitives as far as possible, having a regular DataFrame wrapping them should work fine.

[TomAugspurger]

What are the arguments for keeping SparseDataFrame around? I'm not familiar enough to say for sure whether this is possible, but ideally we would

• Clearly document that DataFrame can hold sparse data
• Move all current sparse-specific methods to a .sparse accessor (density, to_coo). Some of these methods would error if the DataFrame isn't entirely sparse.
• Figure out how to handle default_fill_value
• Deprceate SparseDataFrame in favor of a DataFrame holding sparse arrays

What can a SparseDataFrame do that a DataFrame[SparseArray] can't?

[xpe]

I hope we'd all agree that support for sparsity is very useful. I don't know the internals of Pandas, so I won't weigh in on the particular question of "What can a SparseDataFrame do that a DataFrame[SparseArray] cannot?"

As a user, I'll share my use case and an appreciation for clear documentation.

My use case involves load_svmlight_file from scikit-learn:

from sklearn.datasets import load_svmlight_file
features, labels = load_svmlight_file(data_filename)  # sparse data
feature_labels = ['a', 'b', 'c']  # etc
df = pd.SparseDataFrame(data=features, columns=feature_labels)
df.plot(y='c')  # fails
df.to_dense().plot(y='c')  # succeeds

I'd be open to using other ways to load the sparse LibSVM data, but I don't currently know them. Any recommendations?

Having .plot() work in this case with SparseDataFrame would be nice. :)