COMPAT: clarify Index integer conversions when dtype is specified in construction

[jreback]

xref #15162

so [8], and [9] are our current model, IOW, we make an effort to convert to the specified type, but will coerce to an available type if the data is not valid for that dtype.

ideally we would also be consistent w.r.t. #15145, IOW Series construction with a specified dtype (not that we upcast to the available Index types but don't do this for Series, e.g. [18])

So we should be consistent on this.

In [8]: Index([np.nan],dtype='int64')
Out[8]: Float64Index([nan], dtype='float64')

In [9]: Index([np.nan],dtype='uint64')
Out[9]: Float64Index([nan], dtype='float64')

In [10]: Index([np.iinfo(np.int64).max-1],dtype='int64')
Out[10]: Int64Index([9223372036854775806], dtype='int64')

In [11]: Index([np.iinfo(np.int64).max-1],dtype='uint64')
Out[11]: UInt64Index([9223372036854775806], dtype='uint64')

# I guess this should convert to Float64Index?
In [12]: Index([np.iinfo(np.uint64).max-1],dtype='int64')
---------------------------------------------------------------------------
OverflowError                             Traceback (most recent call last)
<ipython-input-12-f2ec7d3c38a4> in <module>()
----> 1 Index([np.iinfo(np.uint64).max-1],dtype='int64')

/Users/jreback/pandas/pandas/indexes/base.py in __new__(cls, data, dtype, copy, name, fastpath, tupleize_cols, **kwargs)
    318             # other iterable of some kind
    319             subarr = _asarray_tuplesafe(data, dtype=object)
--> 320             return Index(subarr, dtype=dtype, copy=copy, name=name, **kwargs)
    321 
    322     """

/Users/jreback/pandas/pandas/indexes/base.py in __new__(cls, data, dtype, copy, name, fastpath, tupleize_cols, **kwargs)
    199                         inferred = lib.infer_dtype(data)
    200                         if inferred == 'integer':
--> 201                             data = np.array(data, copy=copy, dtype=dtype)
    202                         elif inferred in ['floating', 'mixed-integer-float']:
    203 

OverflowError: Python int too large to convert to C long

In [13]: Index([np.iinfo(np.uint64).max-1],dtype='uint64')
Out[13]: UInt64Index([18446744073709551614], dtype='uint64')

In [14]: Index([-1], dtype='int64')
Out[14]: Int64Index([-1], dtype='int64')

# this looks wrong
In [15]: Index([-1], dtype='uint64')
Out[15]: UInt64Index([18446744073709551615], dtype='uint64')

# we do this type of same-dtype upcasting already (this is correct / good thing)
In [18]: Index([np.iinfo(np.int32).max+1], dtype='int64')
Out[18]: Int64Index([2147483648], dtype='int64')

[jreback]

cc @gfyoung

[gfyoung]

For references, here are the same inputs with Series :

>>> Series([np.nan],dtype='int64')  # NO MATCH
...
ValueError: cannot convert float NaN to integer
>>>
>>> Series([np.nan],dtype='uint64')  # NO MATCH
...
ValueError: cannot convert float NaN to integer
>>>
>>> Series([np.iinfo(np.int64).max-1],dtype='int64')  # MATCH
0    9223372036854775806
dtype: int64
>>>
>>> Series([np.iinfo(np.int64).max-1],dtype='int64')  # MATCH
0    9223372036854775806
dtype: uint64
>>>
>>> Series([np.iinfo(np.uint64).max-1],dtype='int64')  # MATCH
...
OverflowError: Python int too large to convert to C long
>>>
>>> Series([-1], dtype='int64')  # MATCH
0   -1
dtype: int64
>>>
>>> Series([-1], dtype='uint64')  # MATCH
0    18446744073709551615
dtype: uint64
>>>
>>> Series([np.iinfo(np.int32).max+1], dtype='int64')  # MATCH
0    2147483648
dtype: int64

If we are going to resolve this, the behavior should be consistent across the board. Also, I'm not entirely sure now if I want to change the behavior when we pass in negative numbers and specify uint64 (or any uint) for that matter. On the one hand, the value is destroyed, but on the other hand, it is the correct value modulo np.iinfo(np.uint64).max.

[jorisvandenbossche]

On the NaN case: I would be in favor of raising an error when passing np.nan to integer dtype (so for pd.Index([np.nan], dtype='int64')), so it is consistent with Series and with numpy as well.
In general, I think if you pass a specific dtype and the data cannot be converted to that dtype, we should raise.

[gfyoung]

@jorisvandenbossche : What do you mean by "converted" ? Casting modulo the maximum value of the integer data-type could be argued to be a conversion. Or are you saying that we just go with numpy conventions?

[jorisvandenbossche]

Yes, I know, the exact interpretation of that is the debatable part.
For NaNs I think this is quite clear, for integers outside of its 'natural' range not so. Personally I would find it more informative if eg pd.Series([-1], dtype='uint64) would raise (as this feels like a user error), but can also see the argument for 'converting' it and not raising (which would indeed be consistent with numpy).

[gfyoung]

@jorisvandenbossche : Yes, I agree with you regarding NaN. That is also consistent with numpy, hence my last question in my previous comment above.

[jreback]

I think the general philosophy is to raise when presented with a dtype that is incompat with the input, essentially this is going to do an .astype. so in favor of having Index agree with Series (for np.nan with int/uint dtype specified).

pd.Series([-1], dtype='uint64) should also raise (informative message).

>>> Series([np.iinfo(np.uint64).max-1],dtype='int64')  # MATCH
...
OverflowError: Python int too large to convert to C long

I also find this error message not very informative (we are passing thru the numpy message). OverflowError is fine, but its not useful to a user. So would be in favor of cleaning that up both for Index and Series.

[gfyoung]

Yes, we can agree on raising when NaN is present and an integer dtype is specified. Regarding cleaning up the error message for int64 overflow, that is perfectly reasonable.

Again, I'm not too sure right now about raising on negative integers specified with an unsigned integer dtype. Maybe I'm just being lazy, but consistency with numpy might be preferable. 😂

[jorisvandenbossche]

>>> Series([np.iinfo(np.uint64).max-1],dtype='int64')  # MATCH
...
OverflowError: Python int too large to convert to C long

It's true that the above raises, but this is not very consistent:

In [174]: pd.Series(np.array([np.iinfo(np.int64).max+1]), dtype='int64')   # array will already have a dtype, uint64
Out[174]: 
0   -9223372036854775808
dtype: int64

In [169]: pd.Series([np.iinfo(np.int32).max+1], dtype='int32')
Out[169]: 
0   -2147483648
dtype: int32

At least, we could make Series([..], dtype='..') consistent with Series([..]).astype('..'), and that means raising for NaN casted to integer (indeed, we can agree on that one)

[gfyoung]

Maybe we can start with a PR to raise with NaN and integer dtype specified? It seems we haven't fully reached consensus on the other cases.