PERF: speed up certain string operations

[jreback]

from SO

on big enough strings this might be quite useful for a number of string ops.

import pandas as pd
import random
import numpy as np
from StringIO import StringIO

def make_ip():
    return '.'.join(str(random.randint(0, 255)) for n in range(4))

df = pd.DataFrame({'ip': [make_ip() for i in range(20000)]})

%timeit df[['ip0', 'ip1', 'ip2', 'ip3']] = df.ip.str.split('.', return_type='frame')
#1 loops, best of 3: 3.06 s per loop

%timeit df[['ip0', 'ip1', 'ip2', 'ip3']] = df['ip'].apply(lambda x: pd.Series(x.split('.')))
#1 loops, best of 3: 3.1 s per loop

%timeit df[['ip0', 'ip1', 'ip2', 'ip3']] = \
    pd.read_table(StringIO(df['ip'].to_csv(None,index=None)),sep='.')
#10 loops, best of 3: 46.4 ms per loop

[cgevans]

Interestingly, my initial thought that this was because Pandas' split just iterates through in Python was wrong: Using the obvious and pure Python [ x.split('.') for x in list(df['ip'])] is actually 400 times faster than using Pandas. Thus there's something else that's causing the problem.

It seems like vast amounts of time are spent in maybe_convert_objects and _possibly_cast_to_datetime, amongst other things. I'll have to look into it further.

[jreback]

it seems that this operation creates a Series then passes to the DataFrame constructor. No need to do this, the list-like operation should effectively be this.

In [33]: %timeit pd.DataFrame([ x.split('.') for x in list(df['ip']) ])
100 loops, best of 3: 13.4 ms per loop

[jreback]

simple change fixes this.

In [8]: %timeit df['ip'].str.split('.',return_type='frame')
100 loops, best of 3: 20 ms per loop

In [9]: quit()
[jreback-~/pandas] git diff
diff --git a/pandas/core/strings.py b/pandas/core/strings.py
index 6e603f6..b8ea27f 100644
--- a/pandas/core/strings.py
+++ b/pandas/core/strings.py
@@ -723,7 +723,7 @@ def str_split(arr, pat=None, n=None, return_type='series'):
             regex = re.compile(pat)
             f = lambda x: regex.split(x, maxsplit=n)
     if return_type == 'frame':
-        res = DataFrame((Series(x) for x in _na_map(f, arr)), index=arr.index)
+        res = DataFrame([x for x in _na_map(f, arr)], index=arr.index)
     else:
         res = _na_map(f, arr)
     return res

[cgevans]

It seems like avoiding the list creation could be done as well, though it may not make much difference, and I'm not quite sure how Pandas handles iterators internally:

diff --git a/pandas/core/strings.py b/pandas/core/strings.py
index 6e603f6..8fb7f10 100644
--- a/pandas/core/strings.py
+++ b/pandas/core/strings.py
@@ -723,7 +723,7 @@ def str_split(arr, pat=None, n=None, return_type='series'):
             regex = re.compile(pat)
             f = lambda x: regex.split(x, maxsplit=n)
     if return_type == 'frame':
-        res = DataFrame((Series(x) for x in _na_map(f, arr)), index=arr.index)
+        res = DataFrame((x for x in _na_map(f, arr)), index=arr.index)
     else:
         res = _na_map(f, arr)
     return res

Ah; it just converts to a list. Oh well.

[sinhrks]

Because this changes current behavior (non-str values are all converted to str), it may be an option to prepare a shortpath for all-string values. On my environment, numpy's string method is faster than above workaround.

%timeit df[['ip0', 'ip1', 'ip2', 'ip3']] = \
    pd.read_table(StringIO(df['ip'].to_csv(None,index=None)),sep='.')
# 10 loops, best of 3: 80.9 ms per loop

%timeit pd.DataFrame(np.core.defchararray.split(df['ip'].values.astype(str), '.'))
# 10 loops, best of 3: 29.9 ms per loop

[cgevans]

sinhrks: the CSV write/read method is a horrible hack. I don't think jrevack's legitimate solution changes current behavior, and it is significantly faster than the hack, likely in line with numpy's performance.

jreback: while I'd be happy to do a PR for this if necessary, I assume you have it dealt with?

[jreback]

pull requests are welcome on this

[sinhrks]

@cgevans Thanks for your cooperation :) What I meant in current behavior is:

s = pd.Series([1.1, '2.2'])

# current behavior (non-strings are left as NaN)
s.str.split('.', expand=True)
#      0    1
# 0  NaN  NaN
# 1    2    2

# numpy (non-strings are forced to be converted)
pd.DataFrame(list(np.core.defchararray.split(s.values.astype(str), '.')))
#    0  1
# 0  1  1
# 1  2  2

# cant use numpy method without changing dtype
pd.DataFrame(list(np.core.defchararray.split(s.values, '.')))
# TypeError: string operation on non-string array

I think numpy funcs can be used when the values are all string or unicode (maybe regular case). One idea is to use pandas.lib.is_string_array for check and use numpy logic if possible.

[cgevans]

@sinhrks, I think what you're showing is behavior in branches for 10085 / 9847 , not the current pydata/master. I don't have those branches, and am not sure where that work is right now. But I think what you keep referring to as changing behavior is the workaround in the first post here, which is not what I'm discussing, and not what jreback is discussing. I'll make a PR momentarily.

With that said, there is a NaN issue that I'm working on addressing, but it's not quite the same.