Support Generic Container Types

[jaheba]

Feature Request

We want to define our own generic container-types (similar to List[T]), but we want to do the type-checking on our own:

class Array(numpy.ndarray, Collection[T]):
    @classmethod
    def get_validators(cls):
        yield cls.validate_type

    @classmethod
    def validate_type(cls, val):
        return np.array(val, dtype=cls.__args__[0])

(We read in very large lists from a json-file, and pydantic's validation logic is a real bottleneck).

class MyModel:
    values: Array[float]

Problem

For now, pydantic assumes that generic types are from a selected set (Tuple, List, Mapping, plus a few others) and therefore this assertion error is thrown here:

assert issubclass(origin, Mapping)

Possible solutions

Allowing this, could generally be done in two ways:

• Treat custom generic types not differently than other types. The type-parameter is just ignored by pydantic.
• Have a notion of general generic types in pydantic, and provide a validation-interface which allows users to interact with the type-parameters.

[samuelcolvin]

Why can't you do this with validators and pre=True, whole=True and type List[float]? Or sequence as per #304.

I definitely think validation via the get_validators route should work even if the class inherits from Collection[T]. Pull request welcome, although I realise this might be quite tricky.

[jaheba]

The issue is that pydantic's validation logic for lists kicks in and make parsing the data ~10x slower:

parse_array: Avg 3.4680049419403076 s
parse_list: Avg 38.27087903022766 s

(Data is ~180MB in size).

[samuelcolvin]

how about using something like

from pydantic import BaseModel, validator
import numpy


class Model(BaseModel):
    values: numpy.ndarray

    @validator('values', pre=True)
    def parse_values(v):
        return numpy.array(v, dtype=float)

    class Config:
        arbitrary_types_allowed = True


m = Model(values=[1, 2, 3])
print(m)

as a work around?

I'll look into generics more in future, but won't be immediate.

[samuelcolvin]

Or to close to what you originally had

class TypedArray(numpy.ndarray):
    @classmethod
    def __get_validators__(cls):
        yield cls.validate_type

    @classmethod
    def validate_type(cls, val):
        return numpy.array(val, dtype=cls.inner_type)

class ArrayMeta(type):
    def __getitem__(self, t):
        return type('Array', (TypedArray,), {'inner_type': t})

class Array(numpy.ndarray, metaclass=ArrayMeta):
    pass

class Model(BaseModel):
    values: Array[float]

m = Model(values=[1, 2, 3])
print(m)

[jaheba]

Thanks for your suggestions!

The first one is a bit unergonomic in my view, since we want to reuse the Array-type a few times at different places.

The second one should work :) That said, I wouldn't be unhappy if pydantic gets improved support for generics in the future.

[samuelcolvin]

I think replaced by #556 which is more succinct. Let me know if that's significantly different.

[bharling]

+1 For this for me - I'm trying to figure out how to support Association Proxy fields from SQLAlchemy and map them to Lists. The end result should be array-like but the Association is an instance of Collection rather than list so throws an assertion error when passed as a value to my pydantic model

class MySAModel(Base):
    children = association_proxy(
        'through_table_relation',
        'child_id',
        creator=lambda child_id: ChildOfParent(child_id=child_id)
    )

    ...

instance = Session().query(MySAModel).first()

instance.children # << [1,2,3]
isinstance(instance.children, typing.Collection) # << True
isinstance(instance.children, typing.List) # << False


class MySerializer(BaseModel):
    children: List[int] = []

    class Config:
        orm_mode = True

MySerializer.from_orm(instance)

>   ???
E   pydantic.error_wrappers.ValidationError: 1 validation error for MySerializer
E   children
E     value is not a valid list (type=type_error.list)

Support for Collection here would be really handy. The second solution using metaclasses above does get round this but the inner type of the container is lost when generating a schema which is unfortunate

[dmontagu]

@bharling if you are willing to get your hands dirty, you can override BaseModel._decompose_class. With a little massaging you should be able to modify the response so that the "children" key's value is converted to a list first.

(Although it starts with _ to prevent field name collisions, this method is intended to be part of the public API.)

I think in v1.0 the GetterDict interface will have improvements to make it easier to apply validators when initializing via from_orm, but you should be able to get this working now without too much effort.

[bharling]

ah ok great thanks for the quick response, I'll see how I get on.

[dmontagu]

@bharling If you want better support around this point, please leave some feedback on #821 or #822