PERF: have series/panel arithmetic operators use expressions (numexpr)

[jreback]

[jtratner]

I have this done for Series and implemented for scalar arithmetic in Panel. (less clear if it's possible / useful to accelerate combination operations with numexpr). @cpcloud @jreback could one of you write up how to create a Panel with >10K items with integer, floats, mixed integer and integers with zeros?

Basically, like these (which are what I'm using for test cases in test_expressions.py currently), but using Panel instead (don't need multiple, just one version each would be fine):

_frame  = DataFrame(np.random.randn(10000, 4), columns = list('ABCD'), dtype='float64')
_frame2 = DataFrame(np.random.randn(100, 4),   columns = list('ABCD'), dtype='float64')
_mixed  = DataFrame({ 'A' : _frame['A'].copy(), 'B' : _frame['B'].astype('float32'), 'C' : _frame['C'].astype('int64'), 'D' : _frame['D'].astype('int32') })
_mixed2 = DataFrame({ 'A' : _frame2['A'].copy(), 'B' : _frame2['B'].astype('float32'), 'C' : _frame2['C'].astype('int64'), 'D' : _frame2['D'].astype('int32') })
_integer  = DataFrame(np.random.randint(1, 100, size=(10001, 4)), columns = list('ABCD'), dtype='int64')

[jreback]

also examples in
pandas.utils.testing.makePanel

In [17]: _frame2 = DataFrame(np.random.randn(100, 4),   columns = list('ABCD'), dtype='float64')

In [18]: df = DataFrame({ 'A' : _frame2['A'].copy(), 'B' : _frame2['B'].astype('float32'), 'C' : _frame2['C'].astype('int64'), 'D' : _frame2['D'].astype('int32') })

In [19]: Panel(dict(ItemA = df, ItemB = (df+1).astype('int64')))
Out[19]: 
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 100 (major_axis) x 4 (minor_axis)
Items axis: ItemA to ItemB
Major_axis axis: 0 to 99
Minor_axis axis: A to D

[jtratner]

@cpcloud FYI: numexpr isn't working with // right now (at least with our containers), produces the following exception within _evaluate_numexpr:

    Exception: TypeError("unsupported operand type(s) for //: 'VariableNode' and 'VariableNode'",)

[cpcloud]

Yup check out my list on the eval thread I think I note it there. Not sure what to do there. What's weird is that they support scalar floor division

[jtratner]

@cpcloud maybe we could special case it to do true division and then convert it to int?

[cpcloud]

Yeah that should be fine.

[cpcloud]

Hm but let me think about it cuz for a complex expression how to cast without performing the op?

[jtratner]

I setup series to use numexpr. Is this speed increase at all useful? Unclear how good it actually would be...

In [10]: ser = Series(np.random.randn(1000000))

In [11]: ser2 = Series(np.random.randn(1000000))

In [12]: %%timeit
   ....: ser * ser2
   ....:
100 loops, best of 3: 6.87 ms per loop

In [13]: expr.set_use_numexpr(True)

In [14]: %%timeit
   ....: ser * ser2
   ....:
100 loops, best of 3: 6.76 ms per loop

In [15]: %%timeit
   ....: ser / ser2
   ....:
100 loops, best of 3: 7.89 ms per loop

In [16]: expr.set_use_numexpr(False)

In [17]: %%timeit
   ....: ser / ser2
   ....:
100 loops, best of 3: 10.4 ms per loop

[jreback]

In reality these are very similar to numpy operations and they prob cannot be parallelized, which is where a large part fo the speedups come from; doing these same operations on frames yields pretty big speedups (and even better when we have eval, which will enable multiple operates to be sent to ne)

[jtratner]

It just occurred to me that I'm using a 2 core Macbook Air, so I might not notice as much of a difference or I'm not using the right test cases to see the difference. (frame appears to be accelerated by about 30%, huge panel very little and series very little)

[jreback]

build the eval branch that @cpcloud is working on

that's interesting because the more things u have in a term the faster it gets

[cpcloud]

@jtratner just fyi as of now only scalar arith ops with series/frame and already-aligned frames and series arith ops work as well...

[cpcloud]

it's also fun to make a huge frame and run htop to see all of my cores being used for ops :)