Support for arbitrary range types

[Manu343726]

first of all, tag this as "feature-request" "discussion" etc, for the first time I'm not coming here with a bug report :P

I've been thinking about reflecting any kind of iterable range type. By range I mean any type where calls to std::begin() and std::end() are valid.

I see two advantages of implementing that interface:

• Out of the box support for any container: As long as you're doing a foreach loop (which is the case in most of the template loops) it works. Of course we could provide a fake size() method that does std::distance(), for the uncommon case an user applies a length filter on the range.

• Lazy ranges: This could be a game changer. By accepting arbitrary range types users could pass lazy ranges to the template, which means true zero-copy API binding. In my case, with @foonathan's cppast, a coroutine range implementing the AST visit comes to my mind.

[flexferrum]

I suppose, you can do it in the same way as ContainerReflector implemented:
https://github.com/flexferrum/Jinja2Cpp/blob/a146f5bdae19d6fe2eff165f03ccf20cde4ddce8/include/jinja2cpp/reflected_value.h#L114

Actually, I thought about generic range reflectors. But I postponed it till the main Jinja2 features would be implemented and debugged. And you should always keep in mind objects lifetime and prevent situation you run across recently. With arbitrary/generator that's even easier than with temporary containers.

The second point: good implementation of this feature requires refactoring of the current for statement implementation and GenericList data type. It should be closer to the bare Jinja2 when loop 'knows' is it necessary to calculate size of the iterable range and so on. And iterates it not by the index, but by the iterator.

[Manu343726]

I suppose, you can do it in the same way as ContainerReflector implemented

I'm not sure, since you're handling the range there by index. The point of lazy ranges is that the iterator is visited only once, with the iteration going from begin to end. I would instead write a generic range for iteration adapter (For the {% for item in range %} loops). Something like the boost::range::any_range, but that type-erases the value type too.

[Manu343726]

Just figured out a way: With range-v3 we could use a transform view to take any range value and transform it into a jinja2::Value:

template<typename Range>
ranges::v3::any_view<jinja2::Value> make_type_erased_range(const Range& range)
{
    return {range | ranges::v3::transform([](const auto& item)
    {
        return jinja2::Reflect(item);
    })};
}

Now you can wrap a ranges::v3::any_view<jinja2::Value> in a generic range interface:

[flexferrum]

I'm not sure, since you're handling the range there by index.

I see no easy way to avoid this till Jinja2 lists/tuples should support both index-based and range-based access concept. From another point of view, lists in bare Jinja2 could be generator-based (which can cause tricky errors in loops according to their issues tracker). The one of possible ways to allow index-based access only if underlying collection is support it.

Now you can wrap a ranges::v3::any_viewjinja2::Value in a generic range interface:

You missed the interface definition. :) Personally I tends to implement .Net-based collection access style. I.e. introduce something like IEnumerable/IEnumerator interface which is (usually) quite simple. And then implement it for any type of collections/ranges.

[Manu343726]

Here comes the full example of what I think Jinja2Cpp could be capable of with this generic range support:

template<typename AstEntity>
using coro_t = boost::coroutine2::coroutine<type_safe::object_ref<const AstEntity>>;

template<typename AstEntity>
using yield_t = coro_t<AstEntity>::push_type;

template<typename AstEntity, typename Predicate>
void visit(yield_t<AstEntity>& yield, const cppast::cpp_entity& root, Predicate predicate)
{
    cppast::visit(root, [&yield, predicate](const cppast::cpp_entity& entity, const cppast::cpp_visitor_info& info)
    {
        if(entity.kind() == AstEntity::kind() && predicate(root, entity, info))
        {
            yield(type_safe::cref(entity));
        }
    });
}

for(const auto& class_ : visit<cppast::cpp_class>(parser.parse("foo.hpp"), [](const auto& root, const auto& entity, const auto& info)
{
    return entity.name() == "MyClass";
}))
{
    std::cout << "Another class named \"MyClass\" found!\n";
}

template<>
struct TypeReflection<cppast::cpp_file>
{
    ...
    {"classes", [](const cppast::cpp_file& file) { return jinja2::Reflect(visit<cppast::cpp_class>(file, ...)); }}
    ...
};

[Manu343726]

You missed the interface definition. :)

I think that's my main problem, that I've tried to see the full example of how a generic interface could be implemented and used to do TypeReflection for my custom container/range type. I think once I see it in full we will agree 100%.

Personally I tends to implement .Net-based collection access style. I.e. introduce something like IEnumerable/IEnumerator interface which is (usually) quite simple. And then implement it for any type of collections/ranges.

That's perfectly fine for me, since it's basically a different POV of the same problem. Whether the range is accessed by a pair of iterators or by a Iterable interface is not the issue I think (Both could be translated from one to the other by an adapter).

[Manu343726]

and used to do TypeReflection for my custom container/range type. I think once I see it in full we will agree 100%.

After reading your response in #67, now I see that the solution is not TypeReflection but implementing the container interface and specializing the reflector in jinja2::detail namespace to use that interface to access the ranges.

[flexferrum]

Yep.

[flexferrum]

That's perfectly fine for me, since it's basically a different POV of the same problem. Whether the range is accessed by a pair of iterators or by a Iterable interface is not the issue I think (Both could be translated from one to the other by an adapter).

I keep in mind this feature, but due to other activities (and issues) I'm planning to start implementing it not early than the beginning of November. Have to prepare for the conference talk.

[Manu343726]

Conference talk? Where and when?

[flexferrum]

In Minsk, at the Corehard++. Will introduce my Sutter's metaclasses out-of-compiler implementation.

[Manu343726]

Will introduce my Sutter's metaclasses out-of-compiler implementation

Whoa, that's awesome! It's one of the experiments I wanted to do with tinyrefl. Do you have link to the project or something?

[flexferrum]

Implementation prototype is here: https://github.com/flexferrum/autoprogrammer/tree/metaclasses/src/generators/metaclasses

And here the usage sample:
https://github.com/flexferrum/autoprogrammer/tree/metaclasses/test/metaclasses

[Manu343726]

I've been reading your sources in depth to try to implement this myself, but I've found that the for evaluation is much more complex than I thought :(