Proposal: Generalize `Callable` to be able to specify argument names and kinds

[sixolet]

Right now you can specify callables with two patterns of arguments (shown here by example):

• Callable[..., int] takes in any arguments, any number.
• Callable[[int, str, bool], int] takes in a predetermined number of required positional arguments, none of which have names specified.

These don't cleanly match the actual types of callable objects in Python. Argument names, whether arguments are optional, and whether arguments are *args or **kwargs do affect the type of a callable. We should be able to spell these things in the type language. Doing so would enable us to correctly write the types of callback functions, for example.

Callable should take two arguments: an argument list and a return type. The return type is exactly as currently described in PEP484. The argument list is either:

• ..., indicating the function can take any arguments at all.
• Square brackets around a comma separated series of argument specifiers (argspec for short), indicating particulars about the functions arguments

An argument specifier is one of:

• A bare type TYP. This has the same meaning as Arg(TYP)
• Arg(type, name=None), indicating a positional argument. If the name is specified, the argument must have that name.
• OptionalArg(type, name=None), indicating an optional positional argument. If the name is specified, the argument must have that name. (alternate name possibility OptArg(type, name=None)
• StarArg(type), indicating a "star argument" like *args
• KwArg(type), indicating a "double star argument" like **kwargs. (an alternate name here would be Star2Arg(type).

The round parens are an indication that these are not actual types but rather this new argument specifier thing.

Like the rules for python function arguments, all positional argspecs must come before all optional argspecs must come before zero or one star argspecs must come before zero or one kw argspecs.

This should be able to spell all function types you can make in python by defining single functions or methods, with the exception of functions that need SelfType to be properly specified, which is an orthogonal concern.

Some statements I think are true:

• Callable[[Arg(T1, name='foo'), Arg(T2, name='bar')], R] is a subtype of Callable[[T1, T2], R]
• Callable[[T1, OptionalArg(T2)], R] is a subtype of Callable[[T1], R]
• Callable[[StarArg(T1)], R] is a subtype of Callable[[], R] and is also a subtype of Callable[[T1], R] and is also a subtype of Callable[[T1, T1], R] and so on.
• Callable[[T1, StarArg(T1)], R] is a subtype of Callable[[T1], R] and is also a subtype of Callable[[T1, T1], R] and so on, but is not a subtype of Callable[[], R]
• If we want to be able to spell overloaded function types we'll need a specific way of combining callable types in the specific way overloading works; this proposal doesn't address that.

[JukkaL]

Another thing to consider is keyword-only arguments in Python 3. Keyword-only arguments can be optional or non-optional. Maybe use Arg(t, name='foo', keyword_only=True), and similarly for OptionalArg?

[sixolet]

@JukkaL Yep seems good.

[ilevkivskyi]

@sixolet Thankyou for your proposal!
I proposed a similar idea in #239 based on inspect.Signature and Guido rejected it in favor of function stubs. His argument (and I agree with him) is readability. Compare:

def cb_stub(x: T, *args, option: str='verbose') -> int:
    raise NotImplementedError

def func(callback: Callable[cb_stub]):
    ...

and

Args = [Arg(T, name='x'), StarArg(Any), OptionalArg(str, name='option')]

def func(callback: Callable[Args, int]):
    ...

I think the first one is more readable and allows to quickly grasp the function signature. Also it is not clear from your proposal how to type annotate a decorator that preserves the types of all arguments in *args (propbably you also need to discuss how this interacts with variadic generics etc).

I would like to reiterate, function stubs together with OtherArgs discussed in #239 cover vast majority of use cases, so that I think practicality beats purity here.

@JukkaL what do you think about the magic (variadic) type variable OtherArgs? I copy here use cases in decorators from previous discussions:

from typing import OtherArgs, TypeVar, Callable
R = TypeVar('R')

def change_ret_type(f: Callable[[OtherArgs], R]) -> Callable[[OtherArgs], int]: ...

def add_initial_int_arg(f: Callable[[OtherArgs], R]) -> Callable[[int, OtherArgs], R]: ...

def fix_initial_str_arg(f: Callable[[str, OtherArgs], R]) -> Callable[[OtherArgs], int]:
    def ret(*args, **kwargs):
        f('Hello', *args, **kwargs)
    return ret

[sixolet]

Regarding OtherArgs, I think it's separable and so we should consider it separately; being able to spell argument names and kinds is useful in the absence of such a thing. I'll write it up separately once we know what the fate of variadic type variables is likely to be (since the exact definition/meaning of OtherArgs depends on whether we can explain it with variadic typevars).

[sixolet]

It's true that this proposal is similar to your proposal based on inspect.Signature, but it differs in a few important ways:

• It's more concise and readable (I know, a "simple matter of syntax", but I think it actually makes a difference here). It's concise enough that you're likely to spell the whole type of a callable in just one line.
• It's exactly the same as the current syntax in the common/simple case. This also aids readability.
• It's specialized to the purpose of specifying what arguments are like, instead of being intended for runtime introspection and dynamic function creation/metaprogramming. "Do one job well" is UNIX philosophy, not python philosophy, but I think it applies here.

[JukkaL]

@ilevkivskyi I agree with @sixolet that OtherArgs should be discussed separately from this. This proposal (or the alternative proposed syntax) would be useful as such. Callbacks with other than positional arguments have been a pretty common mypy feature request.

[ilevkivskyi]

OK, if you want to proceed with this, I would propose to try to make it more concise. Here are some tweaks, apart from already mentioned OpArgs:

• make name a first argument,
• it could be positional,
• no one probably will use Arg(T) it is just equivalent to T, so that better make the type optional:

Arg('x', T)
Arg('y') # same as Arg('y', Any)

• probably use shorter keyword for keyword only kw_only=True,
• allow omitting kw_only in obvious places, e.g., after StarArg,
• missing type in StarArg and KwArg should also mean Any (like actually everywhere in PEP 484, missing type means Any).

With all these tweaks it will look like this:

def fun1(x: List[T], y: int = 0, *args, **kwargs: str) -> None:
    ...

has type

Callable[[Arg('x', List[T]), OptArg('y', int), StarArg(), KwArg(str)], None]

Second example:

def fun2(__s: str, __z: T, *, tmp: str, **kwargs) -> int:
    ...

has type

Callable[[str, T, Arg('tmp', str, kw_only=True), KwArg()], int]

It looks more concise, and still readable. What do you think?

[sixolet]

@ilevkivskyi I like most of that.

The only bit I think I disagree with is

allow omitting kw_only in obvious places, e.g., after StarArg

And that's because I think in this particular case consistency is more important than brevity.

Huh. Can we omit the parens when they're not being used for anything? Why not

Callable[[str, T, Arg('tmp', str, kw_only=True), KwArg], int]

[sixolet]

(Answering my own question, the round parens remind us it's an argspec not a type. Right.)

[ilevkivskyi]

@sixolet OK
Now that you have +1 from Jukka, you should get an approval from Guido and then add this to the PEP and typing.py.

By the way I was thinking a bit about how this will interoperate with variadic generics, and I think the solution is simple: StarArg and KwArg should be allowed to accept them as an argument and that's it. Then you could easily type a decorator with something like

As = TypeVar('As', variadic=True)
Ks = TypeVar('Ks', variadic=True)

Input = Callable[[int, StarArg(As), KwArg(Ks)], None]
Output = Callable[[StarArg(As), KwArg(Ks)], None]
Deco = Callable[[Input], Output]

Moreover OtherArgs could be easily defined as [StarArg(As), KwArg(Ks)], so that one can write (unpacking just unpacks two items here):

Input = Callable[[int, *OtherArgs], None]
Output = Callable[*OtherArgs], None]

But now it is probably not needed in the specification. So that now I give your proposal a solid +1.

[elazarg]

I don't understand what's wrong with something like Callable[lambda a: int, *, b: str =...:...]

It will be much easier for type checkers to learn, and the meaning is obvious. Why invent new ways of saying the same thing?

[JukkaL]

It's not syntactically valid. Lambdas can't have type annotations.

[sixolet]

@Lazarg Also, we should be careful to avoid confusing people regarding "how do you write a function" vs "how do you write the type of a function"