implement partial aggregates (LArray.regroup and Axis.regroup)

[gdementen]

Implement an easier way to aggregate only part of an axis and leave other labels intact:

>>> a = ndtest(10)
>>> a.sum('a0;a1..a3 >> a13;a4;a5;a6;a7;a8;a9')
a  a0  a13  a4  a5  a6  a7  a8  a9
    0    6   4   5   6   7   8   9

[gdementen]

Technically, that should not be too hard (*), but I am unsure about the syntax:

>>> a.partial_agg(sum, 'a1..a3 >> a13')
>>> a.partial_sum('a1..a3 >> a13')
a  a0  a13  a4  a5  a6  a7  a8  a9
    0    6   4   5   6   7   8   9
>>> a.partial_mean('a1..a3 >> a13')
a   a0  a13   a4   a5   a6   a7   a8   a9
   0.0  2.0  4.0  5.0  6.0  7.0  8.0  9.0

(*) either create the group explicitly like above, or split the array using LArray.split()

[gdementen]

Note that we must also support arbitrary (non-contiguous) groups and (maybe) overlapping groups, which will make an implementation via .split()/.chunks() mostly impossible:

>>> a.partial_sum('a1..a3 >> a13;a6..a8 >> a68')
a  a0  a13  a4  a5  a68  a9
    0    6   4   5   21   9
>>> a.partial_sum('a1,a3,a4 >> a134;a6,a8 >> a68')
a  a0  a134  a2  a5  a68  a7  a9
    0     8   2   5   14   7   9

[gdementen]

Now that I think of it, it might be better to implement this as a method on Axis, so that we do not have to define extra aggregate methods and it works out of the box for any aggregate. The difficulty in that case is to find a good name for the method:

>>> arr.a.partial('a1,a3,a4 >> a134;a6,a8 >> a68')
(a['a0'],
 a['a1', 'a3', 'a4'] >> 'a134',
 a['a2'],
 a['a5'],
 a['a6', 'a8'] >> 'a68',
 a['a7'],
 a['a9'])
>>> arr.sum(arr.a.partial('a1,a3,a4 >> a134;a6,a8 >> a68'))
a  a0  a134  a2  a5  a68  a7  a9
    0     8   2   5   14   7   9

This seems technically interesting, but not very readable/obvious what it means.

[gdementen]

I don't think partial is clear enough. Is partial_grouping understandable enough?

>>> arr = ndtest(10)
>>> a = arr.a
>>> a.partial_grouping('a1,a3,a4 >> a134;a6,a8 >> a68')
(a['a0'],
 a['a1', 'a3', 'a4'] >> 'a134',
 a['a2'],
 a['a5'],
 a['a6', 'a8'] >> 'a68',
 a['a7'],
 a['a9'])
>>> arr.sum(a.partial_grouping('a1,a3,a4 >> a134;a6,a8 >> a68'))
a  a0  a134  a2  a5  a68  a7  a9
    0     8   2   5   14   7   9

[gdementen]

maybe Axis.regroup() ?

[gdementen]

When groupby is done, we will be able to do this via set_labels + groupby. That would be an improvement compared to the current situation, but maybe not good enough as it is still quite verbose and inefficient.

>>> arr = ndtest(10)
>>> arr.set_labels('a', {'a1': 'a134', 'a3': 'a134', 'a4': 'a134', 'a6': 'a68', 'a8': 'a68'}).groupby('a').sum()
a  a0  a134  a2  a5  a68  a7  a9
    0     8   2   5   14   7   9

[gdementen]

Other ideas:

• support passing groups to set_labels (the last example requires allow omitting axis (guess_axis) for set_labels #634)

>>> arr = ndtest(10)
>>> # I like this, because it simply generalize what we already have. We might want to implement this regardless of this "partial grouping" feature
>>> arr.set_labels('a', {X.a['a1,a3,a4']: 'a134', X.a['a6,a8']: 'a68'})).groupby('a').sum()
a  a0  a134  a2  a5  a68  a7  a9
    0     8   2   5   14   7   9
>>> # or even reuse an existing group label (this might be going too far?)
>>> arr.set_labels('a', (X.a['a1,a3,a4'] >> 'a134', X.a['a6,a8'] >> 'a68')).groupby('a').sum()
a  a0  a134  a2  a5  a68  a7  a9
    0     8   2   5   14   7   9
>>> # ... but this would be practical
>>> arr.set_labels('a1,a3,a4 >> a134;a6,a8 >> a68').groupby('a').sum()
a  a0  a134  a2  a5  a68  a7  a9
    0     8   2   5   14   7   9

• use a keyword argument to aggregate functions (unsure about the name of the kwarg though)

>>> arr.sum('a1,a3,a4 >> a134;a6,a8 >> a68', partial_agg=True) # or "partial" or "keep_other" or ...
a  a0  a134  a2  a5  a68  a7  a9
    0     8   2   5   14   7   9

This gets awkward when we want to combine partial and non partial aggregates.

• use a "modifier attribute" to aggregate functions

>>> arr.partial.sum('a1,a3,a4 >> a134;a6,a8 >> a68')
a  a0  a134  a2  a5  a68  a7  a9
    0     8   2   5   14   7   9

• use a "regroup" method on LArray (probably in combination with Axis.regroup)

>>> arr.regroup('a1,a3,a4 >> a134;a6,a8 >> a68').sum()
a  a0  a134  a2  a5  a68  a7  a9
    0     8   2   5   14   7   9
>>> groups = arr.a.regroup('a1,a3,a4 >> a134;a6,a8 >> a68')
>>> arr.sum(groups)
a  a0  a134  a2  a5  a68  a7  a9
    0     8   2   5   14   7   9

This is my currently preferred option, but (the way I see it) would benefit from a Grid class LArray.regroup() would return such an object. This Grid thing is more or less implemented in my local branch to implement #635.