PERF: additonal GIL releasing

[jreback]

after #10199 (all of the groupbys)
these require slightly different templates as the object cannot have the with nogil:. These are all in pandas/src/generate_code.py

- [ ] groupby_count
- [ ] is_monotonic

• take_1d
• take_2d
• take_2d_multi
• pad
• backfill
• diff
• join_indexer (outer/inner/left)

in algos.pyx

• kthsmallest/kthlargest
• group_median
• roll_*
• drop_duplicates machinery
• nancorr

[mrocklin]

I've run up against drop_duplicates a bit recently. I'd bump that slightly in priority here. No specific rush though.

[jreback]

ints

In [41]: np.random.seed(1234)

In [42]: N = 1000000

In [43]: df = DataFrame({'A' : np.random.randint(0,100,size=N)})

In [44]: def f():
   ....:     for i in range(2):
   ....:         df.drop_duplicates()
   ....:         

In [45]: @test_parallel(2)
   ....: def g():
   ....:     df.drop_duplicates()
   ....:     

In [46]: %timeit f()
10 loops, best of 3: 86.3 ms per loop

In [47]: %timeit g()
10 loops, best of 3: 54.8 ms per loop

objects

In [48]: idx = tm.makeStringIndex(100)

In [49]: df = DataFrame({'A' : idx.take(np.random.randint(0,100,size=N))})

In [50]: %timeit f()
10 loops, best of 3: 244 ms per loop

In [51]: %timeit g()
10 loops, best of 3: 168 ms per loop

@mrocklin what dtypes are you typically dropping? note that dropping on categoricals is particularly efficient (though the conversion cost can be higher than the time simply to drop, but if its already categorized then its cheap).

[mrocklin]

I was dropping integers and categoricals. Does this benefit increase beyond two cores?

[jreback]

In [10]: N = 10000000

In [11]: df = DataFrame({'A' : np.random.randint(0,1000,size=N)})

In [12]: def f():
   ....:     for i in range(8):
   ....:         df.drop_duplicates()
   ....:         

In [13]: @test_parallel(8)
   ....: def g():
   ....:     df.drop_duplicates()
   ....:     

In [14]: %timeit f()
1 loops, best of 3: 2.49 s per loop

In [15]: %timeit g()
1 loops, best of 3: 1.02 s per loop

[jreback]

#10917

will make this:

In [5]: %timeit f()
1 loops, best of 3: 1.6 s per loop

In [6]: %timeit g()
1 loops, best of 3: 595 ms per loop

[jbrockmendel]

This has likely been pushed as far as it’ll go.

[jreback]

can u update the metrics just to see