Possible to distinguish between Sequence[str]/Iterable[str] and str?

[jtatum]

If a function expects an iterable of strings, is it possible to forbid passing in a string, since strings are iterable? Seems like there are many cases where this would be an error, but I don't see an obvious way to check 't','h','i','s'.

[gvanrossum]

Since str is a valid iterable of str this is tricky. Various proposals have been made but they don't fit easily in the type system.

[vedgar]

I think type should never lie, even if it is a white lie. Either we should remove str.__iter__ (or make it yield something else than strs), or we should allow passing 'abc' into a function expecting Iterable[str]. Of course, I'm for second option.

Or we should have a special type name for "iterable of strings that is not a string". Strings are already special, as AnyStr shows. But although AnyStr is able to be represented using more primitive operations, I think it's too early to introduce a "type difference" operation in general. E.g. co(ntra)variance seems weird in that case.

[gpshead]

The problem I have with allowing Sequence[str] or Iterable[str] to be satisfied by str is that the problem of passing a str in where a sequence of (generally non single character) strs is really intended is a common API misuse that a type checker needs to be able to catch.

People can over-specify their APIs by requiring List[str] or Tuple[str] as input instead of the more general sequence or iterable but this is unnatural when teaching people how to type annotate. We'd prefer to just tell everyone to always prefer Iterable or Sequence on input.

Random thought: Would it be possible for our "magic" Text type to lose it's __iter__? So that Iterable[Text] works as desired and forbids a lone str argument?

[ssbr]

It requires more work on the part of API authors, but one option that might be less of a lie is to be able to delete an overload. C++ has a similar problem, where a type being passed in might "work" but you want to forbid it. So they implemented a special overload that, if matched, causes an error. See e.g. this SO thread.

Then one could define the API for Iterable[str], and delete the overload for str.

[gvanrossum]

Mypy doesn't currently have a way to remove methods in a subclass, because it would fail Liskov. But there's a hack possible. You can change the signature of a method override in a way that violates Liskov, and then add a # type: ignore to prevent mypy from complaining. Mypy will then check uses according to the override! So maybe something like this (untested) could be made to work:

class Text(Sequence[str]):
    def __iter__(self) -> None: ...  # type: ignore

[ilevkivskyi]

It actually doesn't work. Because currently there is a rule in mypy: "nominal first" (for various important reasons), if something works using nominal subtyping, then mypy just uses it. In this case Text is still a nominal subtype of Sequence[str].

[gvanrossum]

Hm... Maybe Text could be a Protocol that has the same methods as Sequence except for one?

[ilevkivskyi]

Maybe, TBH I am still not sure what are the costs/benefits here. I am afraid making such big changes in typeshed can break many existing code. But on the other hand if someone wants to do this "locally" it should be a fine solution.

[JukkaL]

A relatively simple approach would be to special case str vs. Iterable[str] / Sequence[str] compatibility in a type checker. This behavior could be enabled through a strictness option. This issue seems quite specific to str (and unicode) so anything more drastic may not be worth it.

[Michael0x2a]

If we assume the type checker has reasonable good dead code analysis capabilities, we could get a solution that's pretty similar to the one C++ has for free by combining @overload and NoReturn. For example:

from typing import Iterable, overload, NoReturn

@overload
def foo(x: str) -> NoReturn: ...
@overload
def foo(x: Iterable[str]) -> int: ...
def foo(x):
    if isinstance(x, str):
        raise Exception()
    else:
        return 1

def main() -> None:
    x = foo(["hello", "world"])
    reveal_type(x)

    # Maybe report dead code?
    y = foo("hello")
    reveal_type(y)

That said, idk if any type checkers actually do handle this case gracefully. Mypy, for example, will just silently ignore the last reveal_type (and warn that y needs an annotation).

Maybe to help this analysis, we could add some sort of ShouldNeverBeEncountered type? Type checkers could add a special-case that reports an error whenever they see some function call evaluates to this type, but otherwise treat it as being identical to NoReturn.

It's not a perfect solution since there's still no definitive way of telling if an Iterable[str] is a str or not, but it'd at least give library authors a way to catch some of the more obvious misuses w/o requiring their users to use a special Text-like protocol.

[gpshead]

Given the norm for most APIs is to accept the iterable and never want plain str we should aim to support that as a trivial annotation that doesn't involve multiple defs and overloading. the oddball situation where someone wants to accept the iterable and plain str should be the complicated one if complexity is needed.

[vedgar]

I think we're trying to expand type hints beyond their original purpose, and it shows.
If I say a_string.rstrip('abc'), the function is going to work perfectly. It will, according to its specification, produce a "copy" of a_string, from which all as, bs and cs are removed at the end. There isn't going to be any "hidden type errors", "accidental mechanisms" or "unintended consequences" that the type hints are usually trying to prevent. Mypy has nothing to do here. 'abc' is just a compact way to write an iterable of strs, that yields 'a', 'b' and 'c' in that order, and then stope.
What you're now trying to do is go beyond "do types match" (they do, absolutely) into "did the caller really intend to write this". And that is a dangerous crossing of responsibility boundaries. Yes, there is a sentence in PEP 484 about mypy being "a powerful linter", but I really think noone wanted mypy to take over all responsibilities of a linter. At least I hope so.
In short: is passing a str as an Iterable[str] a common error? I don't know (in my experience it is not, but of course you have more experience). Does it need to be flagged by a linter? Probably. Are type hints the right way to catch it? NO.

[JukkaL]

Maybe Text could be a Protocol that has the same methods as Sequence except for one?

Unfortunately this would make Text incompatible with str and would generally break typeshed and existing annotations.

[martindemello]

I like the idea of special-casing strings in the tool rather than in the type system, since as @gvanrossum notes, str is an iterable of str (turtles all the way!). Also this sort of type-aware linting is a neat idea, and could be done relatively easily within the typechecker because we have all the information at hand.

[gvanrossum]

all the information at hand

Do we? When I see a function that takes an Iterable[str] or Sequence[str] -- how do we know it is meant to exclude str? Or do we just assume it is always excluded?