BUG: COO matrices are copied and converted

[memeplex]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

Doesn't apply.

Problem description

The documentation for DataFrame.sparse.from_spmatrix() in [1] states that:

All sparse formats are supported, but matrices that are not in COOrdinate format will be converted, copying data as needed.

But the first thing it does [2] is:

data = data.tocsc()

so conversion. And the implementation for tocsc [3] does:

            indptr = np.empty(N + 1, dtype=idx_dtype)
            indices = np.empty_like(row, dtype=idx_dtype)
            data = np.empty_like(self.data, dtype=upcast(self.dtype))

            coo_tocsr(N, M, self.nnz, col, row, self.data,
                      indptr, indices, data)

so copy.

Given that the "native" format is CSC, not COO, it seems weird to me that the documentation insists on COO.

Expected Output

Doesn't apply.

Output of pd.show_versions()

Referenced documentation and code is for 1.3.2.

---

[1] https://pandas.pydata.org/pandas-docs/stable/user_guide/sparse.html
[2] https://github.com/pandas-dev/pandas/blob/v1.3.2/pandas/core/arrays/sparse/accessor.py#L228-L283
[3] https://github.com/scipy/scipy/blob/v1.7.1/scipy/sparse/coo.py#L330-L370

[memeplex]

The documentation may be referring to to_coo but if that's the case it's quite non evident:

All sparse formats are supported, but matrices that are not in COOrdinate format will be converted, copying data as needed. To convert back to sparse SciPy matrix in COO format, you can use the DataFrame.sparse.to_coo() method:

Here the second sentence is introducing to_coo and talking about "convert[ing] back to sparse" as if the previous sentence were about converting from sparse.

Besides it's not clear to me in what sense to_coo doesn't copy since it's np.concatenate'ing lists of cols, rows and values.

[memeplex]

Also, by inspecting the code and the documentation one gets the impression that originally the target format might have been COO but that CSC is more naturally suited for a dataframe, so the documentation may be outdated and the code shows mixed criteria in that it transforms the input to CSC but then offers to go back to COO. Now, the repeat in cols.append(np.repeat(col, len(row))) in to_coo wouldn't have been necessary in a potential to_csc, again stressing that the natural fit to dataframe is CSC. And given the current implementation, one can think of a to_csc method that indeed doesn't copy at all when the dataframe is entirely backed by a CSC matrix (a reference to the original matrix could just have been kept at initialization time). It's rather confusing.

[simonjayhawkins]

Code Sample, a copy-pastable example

Doesn't apply.

maybe a code sample would improve engagement on this issue. (and could be used as a test for a fix)

[memeplex]

I don't know exactly what you expect. I've shown where it happens in the code because these are implementation issues that seem to contradict what's documented. I could provide an example in which modifying one representation won't affect the other one, but this is a bit contrived for a COO representation since it doesn't support item assignment and anyway it's not strictly the point.

[mzeitlin11]

I think CSC format is necessary (so conversion is necessary, at least with current impl) since we access the indices attribute. Perhaps just the user guide should be amended then? The actual docstring for that function actually does correctly single out that Must be convertible to csc format.