API: pd.StringDtype.value_counts should return pd.Int64Dtype

[WillAyd]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

From discussion in https://github.com/pandas-dev/pandas/pull/59330/files#r1695250192

The fact that pd.StringDtype(storage="pyarrow") returns a int64[pyarrow] is a mistake with our API. While this is well intentioned for performance reasons, it introduces an inconsistency with our extension types

If we continue to have pd.StringDtype.value_counts return a pd.Int64Dtype() we can leave it as an implementation detail whether or not that Int64Dtype is backed by pyarrow or numpy. The user interface would not need to change at all; users could simply install (or not install) pyarrow and things should continue to work the same. This would long term also align better with PDEP-13 #58455

Feature Description

Make algorithms for the StringDtype that use pd.NA as the missing value sentinel consistently return other pandas extension types, and leave it as an implementation detail if those are backed by pyarrow or not

Alternative Solutions

status quo

Additional Context

No response

[OmarAli-Ibrahim]

assign it to me

[tan-i-ham]

Hello @WillAyd , I am interested in this task.

I've looked at the code.

My understanding based from current conversation is that, in this test case

pandas/pandas/tests/arrays/string_/test_string.py

Lines 577 to 582 in ecc451d

	if dtype.na_value is np.nan:
	exp_dtype = "int64"
	elif dtype.storage == "pyarrow":
	exp_dtype = "int64[pyarrow]"
	else:
	exp_dtype = "Int64"

The value will become this and continue the test, no matter the storage is "pyarrow" or not.

 if dtype.na_value is np.nan: 
     exp_dtype = "int64" 
 else: 
     exp_dtype = "Int64" 

Is the above understanding correct?

If it is correct, could you guide me to the start of possible code change?
I debug a while, however, not sure where to start to fix the code.
Is in the ArrowExtensionArray.value_counts or value_counts_internal() in pandas/core/algorithms.py? or maybe other?

[WillAyd]

Thanks for the interest / questions. I think should hold off on going too far in this one until the discussion around PDEP-13 gets clearer #58455

Otherwise it is hard to say what should be done here

[tan-i-ham]

Got it. Thanks!