BUG: regression in master for DataFrame.sparse.from_spmatrix

[bmreiniger]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

from scipy.sparse import eye
from pandas import DataFrame
DataFrame.sparse.from_spmatrix(eye(5))

Issue Description

(This bug does not exist on the latest version of pandas, but that checkbox was required to submit the issue.)

In main branch, the example produces a frame with entries 1.0 and NaN (sparse, with fill_value nan).

PR #59064 corrected a bug for other dtypes, but introduced this regression for float dtype.

Expected Behavior

In latest version (2.2.2) and earlier, produces a frame with entries 1.0 and 0.0 (sparse, with fill_value 0).

Installed Versions

INSTALLED VERSIONS

commit : 69fe98d
python : 3.12.4
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19045
machine : AMD64
processor : Intel64 Family 6 Model 140 Stepping 1, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : English_United States.1252

pandas : 3.0.0.dev0+1170.g69fe98dda
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0.post0
pip : 24.0
Cython : None
sphinx : None
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : None
lxml.etree : None
matplotlib : 3.9.0
numba : None
numexpr : None
odfpy : None
openpyxl : None
psycopg2 : None
pymysql : None
pyarrow : None
pyreadstat : None
pytest : 8.2.2
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.14.0
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None

[kushagr96]

take

[kushagr96]

The difference comes from the fact that the default fill value of numpy float64 is 0.0 whereas for pandas it is NaN. I tried changing it for the from_spmatrix function, but it introduces breaking changes for the rest of the Sparse matrix functions.

[bmreiniger]

It's my impression that scipy does everything assuming that the fill value is zero, so that when creating a dataframe from a scipy sparse array pandas should do the same. If that's the case, should this function just override the usual default pandas fill value for float type scipy sparse?

[WillAyd]

This is a tough one. We have two different ways of handling sparse values depending on the underlying type:

1. fill-with-missing-value which we do for floats, complex, datetime and timedelta
2. fill-with-zero-value which we do for ints and bools

fill-with-zero-value could work for ints, floats, and complex (and as you noted, this is how NumPy works), but that same logic would not extend to datetime types, since a 0 value there is still a valid value. On the other hand, we don't support missing values for ints and bools well (at least not using our historical NumPy types, and not without PDEP-16, so fill-with-zero-value has to be used for those

In our current state I don't see any way that we can make everyone happy. In the future, using a missing value for the sparse elements is the only way that could work across all types. So I'm inclined to say we should leave the current behavior as is since it more closely represents where I think we need to be long term, but @mroeschke @jorisvandenbossche @christopher-titchen may have thoughts as well