Type promotion rules

[kgryte]

This issue seeks to come to a consensus on a subset of type promotion rules (i.e., the rules governing the common result type for two array operands during an arithmetic operation) suitable for specification.

As initially discussed in #13, a universal set of type promotion rules can be difficult to standardize due to the needs/constraints of particular runtime environments. However, we should be able to specify a minimal set of type promotion rules which all specification conforming array libraries can, and should, support.

Prior Art

• NumPy: promotion rules follow a type hierarchy (where complex > floating > integral > boolean). See promote_types and result_type APIs and source [1, 2].
• CuPy: follows NumPy's rules, except for zero-dimension arrays.
• Dask: follows NumPy's rules.
• JAX: type promotion table (and source) which deviates from NumPy's promotion rules in two ways: (1) biased toward half- and single-precision floating-point numbers and (2) support for a non-standard floating-point type.
• PyTorch: promotion rules follow a type hierarchy (where complex > floating > integral > boolean) without inspection of value magnitude.
• Tensorflow: requires explicit casting.

Proposal

This issue proposes to specify that all specification conforming array libraries must, at minimum, support the following type promotions:

• floating-point type promotion table:

  	f2	f4	f8
  f2	f2	f4	f8
  f4	f4	f4	f8
  f8	f8	f8	f8

  where

  • f2: half-precision (16-bit) floating-point number
  • f4: single-precision (32-bit) floating-point number
  • f8: double-precision (64-bit) floating-point number

• unsigned integer type promotion table:

  	u1	u2	u4	u8
  u1	u1	u2	u4	u8
  u2	u2	u2	u4	u8
  u4	u4	u4	u4	u8
  u8	u8	u8	u8	u8

  where

  • u1: 8-bit unsigned integer
  • u2: 16-bit unsigned integer
  • u4: 32-bit unsigned integer
  • u8: 64-bit unsigned integer

• signed integer type promotion table:

  	i1	i2	i4	i8
  i1	i1	i2	i4	i8
  i2	i2	i2	i4	i8
  i4	i4	i4	i4	i8
  i8	i8	i8	i8	i8

  where

  • i1: 8-bit signed integer
  • i2: 16-bit signed integer
  • i4: 32-bit signed integer
  • i8: 64-bit signed integer

• mixed unsigned and signed integer type promotion table:

  	u1	u2	u4
  i1	i2	i4	i8
  i2	i2	i4	i8
  i4	i4	i4	i8

Notes

• The minimal set of type promotions outlined above explicitly does not define promotions between types which are not of the same kind (i.e., floating-point versus integer). When converting between types of different kinds, libraries tend to support C type promotion semantics, where floating-point, regardless of precision, has a higher rank/precedence than all integer types; however, they differ in the promoted floating-point precision (e.g., JAX promotes (i8, f2) to f2, while NumPy promotes (i8, f2) to f8). The reason for the discrepancy stems from the particular needs/constraints of accelerator devices, and, thus, by omitting specification here, we allow for implementation flexibility and avoid imposing undue burden.
• Omitted from the above tables are "unsafe" promotions. Notably, not included are promotion rules for mixed signed/unsigned 64-bit integers i8 and u8. NumPy and JAX both promote (i8, u8) to f8 which is explicitly undefined via the aforementioned note regarding conversions between kinds and also raises questions regarding inexact rounding when converting from a 64-bit integer to double-precision floating-point.
• This proposal addresses type promotion among array operands, including zero-dimensional arrays. It remains to be decided whether "scalars" (i.e., non-array operands) should directly participate in type promotion.

[rgommers]

Thanks @kgryte, this looks pretty good to me.

One addition to the prior art may be CuPy. Its type promotion docs bring up a relevant question: do 0-D arrays participate, or are they treated as scalars that cannot cause upcasting?

That scalars don't participate should be added explicitly I think.

[kgryte]

Thanks for mentioning CuPy. Added.

Re: zero-dimensional arrays. Seems to be some disagreement in this regard, as some array libraries have explicitly avoided copying NumPy's exact type promotion behavior, where the magnitude of the scalar is considered when determining the result data type (e.g., CuPy, PyTorch).

IMO, NumPy's behavior is a bit harder to reason about, especially when the scalar value is not known beforehand or is dynamically set and a change in scalar magnitude can result in a different result type.

My current stance is that zero-dimensional arrays are treated just like dimensional arrays when promoting types. For example, if a zero-dimensional array has dtype uint32 and a dimensional array has dtype int16, the result has dtype int64, regardless as to whether the zero-dimensional array scalar value can fit in an 8-bit unsigned integer (uint8). Meaning, zero-dimensional arrays would not be, according to the specification, afforded any special treatment.

For scalar arguments specifically (e.g., add( 5, <array> )), were we to standardize signatures permitting scalar arguments, the specification could require that scalar arguments be treated as zero-dimensional arrays, thereby associated a supported dtype (which could, if not specified, vary across array libraries), and then the promotion rules would apply as normal. This would mean that a scalar argument, when wrapped as a zero-dimensional array, could trigger promotion to a different type.

I think this is the most consistent, and least magical, path forward.

[rgommers]

Hmm, I suspect that will be really bad for usability. It's not just in functions, but also in expressions like 5*x + 3. If you'd have to write that as array(5, dtype=int32) * x + int32(3) to avoid some casting behavior, that would get annoying very quickly.

There's two separate things: NumPy's value-based promotion (which no one wants), and whether scalars participate in type promotion at all (which is a good discussion to have).

[kgryte]

@rgommers Agreed. Maybe three questions then need to be answered:

1. NumPy value-based promotion?
2. Scalar participation in type promotion?
3. Are zero-dimensional arrays considered scalars for the purposes of type promotion?

[rgommers]

Yes makes sense. I'd say "No, No, Dunno".

[shoyer]

3. Are zero-dimensional arrays considered scalars for the purposes of type promotion?

This is definitely the hardest part of this design decision:

(a) Treating 0d arrays like scalars is convenient because it allows for converting Python scalars into 0d arrays without any impact on promotion rules. You can write the equivalent of np.asarray() on all inputs and not worry about non-array objects again.
(b) On the other hand, it introduces an inconsistency between arrays of different ranks. This makes it harder to write (or type check) rank polymorphic code. For example, now add(Array[float32], Array[float64]) -> Array[float64] would fail to type check, because if the float64 input is 0d, it wouldn't participate in type casting rules and the output would be float32.

My personal opinion is that (b) is a fatal flaw with special casing 0d arrays, and there are good alternatives to immediately calling asarray on inputs (e.g., call np.result_type() on inputs to figure out the result type before casting with np.asarray()).

(Perhaps not coincidentally, JAX answers these questions "No, No, No")

[teoliphant]

Generally, I think we should with v1 of the spec be as non-magical as possible recognizing that library authors are our target and that more usable interfaces should and will be written so that end-users won't have some of the difficulties of expression that Ralf suggests.

So I would agree with 1) No and 2) No.

For 3) it seems at first glance like 0d arrays should just be treated as arrays and participate in promotion rules, but the numeric Python community as a long history of 0d arrays and scalars being (mostly) interchangeable and it will be quite hard to not have scalar participation in the type promotion and have 0d array participation. So, I lean to "No" on 3) as well.

Yes, this breaks symmetry on promotion participation at a place that is usually easy to understand (when arr.ndim == 0), rather than the not often specified places where 0d arrays and scalars are treated as equivalent by a given library.

[shoyer]

@rgommers Agreed. Maybe three questions then need to be answered:

1. NumPy value-based promotion?
2. Scalar participation in type promotion?
3. Are zero-dimensional arrays considered scalars for the purposes of type promotion?

I think the wording of (3) is a little confusing because the answer is (possibly?) conditional on the response to (2). E.g., Travis and I both suggested "no" but judging by his language ("will be quite hard to not have scalar participation in the type promotion and have 0d array participation") I think we meant opposite things :)

So let's try this again:

1. NumPy value-based promotion?
2. Scalar participation in type promotion?
3. Zero-dimensional array in participation in type promotion?

My suggestion to these revised questions was "No, no, yes" (all arrays participate in type promotion, matching JAX), but if I interpret Travis's comment correctly he is leaning towards "No, no, no" (only arrays with at least one dimension participate in type promotion)?

(To be clear, by "scalars" I mean Python builtin types int and float, without explicitly encoded precision. None of these array libraries other than NumPy have dtype specific scalars like np.float64, so I would prefer not to encode them in the spec.)

[rgommers]

Zero-dimensional array in participation in type promotion?

Thanks for rephrasing. I think I agree with your "yes" here.

The concern of @teoliphant that I also had I think can be handled with being careful about where one can use scalars. For regular functions, it should be fine to say add(5, x) is forbidden. That doesn't mean 5 + x has to be forbidden (that would be a real pain); if we special-case using Python scalars with __add__, __radd__ et al., then I think we're good.

[teoliphant]

Thanks for clarifying. Yes, I misread the third question. I'm +1 with treating 0-d arrays as arrays and not allowing scalar promotion in function calls themselves -- i.e. keeping the scalar handling rules in the "dunder" methods on any array class.

[kgryte]

Great! Based on feedback, seems there is agreement on (1) not allowing "scalars" to participate in type promotion (e.g., a Python int from triggering type promotion of an array operand and (2) not special-casing zero-dimensional arrays (i.e., zero-dimensional arrays are subject to the same type promotion rules as dimensional arrays).

[rgommers]

This was agreed upon, and the Type Promotion section currently summarizes the 0-D array / scalar issue as:

• Type promotion rules strictly apply when determining the common result type for two array operands during an arithmetic operation, regardless of array dimension. Accordingly, zero-dimensional arrays ar e subject to the same type promotion rules as dimensional arrays.
• Non-array ("scalar") operands are not permitted to participate in type promotion.

That seems in agreement with the discussion here and complete, so closing this issue.