PERF: optimize MultiIndex.from_product

[immerrr]

This PR speeds up MultiIndex.from_product employing the fact that operating on categorical codes is faster than on the values themselves.

This yields about 2x improvement in the benchmark

In [1]: import pandas.util.testing as tm

In [2]: data = [tm.makeStringIndex(10000), tm.makeFloatIndex(20)]

In [3]: %timeit pd.MultiIndex.from_product(data)
100 loops, best of 3: 10.6 ms per loop

In [4]: %timeit pd.MultiIndex.from_arrays(pd.tools.util.cartesian_product(data))
10 loops, best of 3: 23.4 ms per loop

It's only marginally slower in small size cases:

In [1]: data = [np.arange(20).astype(object), np.arange(20)]

In [2]: %timeit pd.MultiIndex.from_product(data)
1000 loops, best of 3: 317 µs per loop

In [3]: %timeit pd.MultiIndex.from_arrays(pd.tools.util.cartesian_product(data))
1000 loops, best of 3: 308 µs per loop

In [4]: data_int = [np.arange(20), np.arange(20)]

In [5]: %timeit pd.MultiIndex.from_product(data_int)
1000 loops, best of 3: 285 µs per loop

In [6]: %timeit pd.MultiIndex.from_arrays(pd.tools.util.cartesian_product(data_int))
1000 loops, best of 3: 269 µs per loop

And this case came as a surprise because the cartesian product is blazingly fast both in old and new versions, but profiling showed that factorization is a lot faster when done on a smaller array:

In [7]: data_large = [np.arange(10000), np.arange(20)]

In [8]: %timeit pd.MultiIndex.from_arrays(pd.tools.util.cartesian_product(data_large))
100 loops, best of 3: 9.88 ms per loop

In [9]: %timeit pd.MultiIndex.from_product(data_large)
100 loops, best of 3: 2.74 ms per loop

[shoyer]

Looks like a nice speedup, but could you please verify that #6439 (cartesian product of a DatetimeIndex) is still fixed?

e.g.:

import pandas as pd
idx = pd.MultiIndex.from_product([[1, 2], pd.date_range('2000-01-01', periods=2)]).values 
print [x.day for _, x in idx]
# should print [1, 2, 1, 2]

In retrospect, I should have added a test for MultiIndex.from_product in #6451.

[jorisvandenbossche]

@shoyer You can always still add a test for that in a new PR

[immerrr]

@shoyer works for me

In [24]: %cpaste
Pasting code; enter '--' alone on the line to stop or use Ctrl-D.
:import pandas as pd
:idx = pd.MultiIndex.from_product([[1, 2], pd.date_range('2000-01-01', periods=2)]).values 
:print [x.day for _, x in idx]
:--
[1, 2, 1, 2]

[jreback]

IIRC their are tests for using DatetimeIndex in MultiIndex.from_product. @immerrr if not enough test coverage, pls add (otherwise ok)

[jreback]

@immerrr looks ok, pls verify test coverage then can merge

[immerrr]

Ok, added the test

[jreback]

ok, ping when green

[immerrr]

good to go

[jreback]

thansk!

[shoyer]

@immerrr Thanks for adding that test!

[immerrr]

@shoyer you're welcome :)