col.replace(dict) takes too much memory

[jankatins]

I've code which run fine about ~10 month ago and now fails due to running out of memory. The code basically does this:

groups = [[1,2,3][222,333,444, 555], ...]  # from a 1.4MB json file
# about 30k such groups, results in ~70k replacements
for replace_list in groups:
        replacement = sorted(replace_list)[0]
        for replace_id in replace_list:
            if replace_id == replacement:
                continue
            replace_dict[replace_id] = replacement
# len(_col) == 974757
_col = df[column].astype("int64")
_col.replace(replace_dict, inplace=True)

I've now split the replacement_dict into 2k chunks and and this works (takes about 20 seconds for each chunk).

[jreback]

can you provide the file, pandas, numpy version that it was better on and what you are trying on now?

also, try wi/o the in-place operation

[jankatins]

The replacement file is no problem, but the dataframe is internal and about 400MB csv unziped :-(

Now I do it similar to this:

groups = [[1,2,3][222,333,444, 555], ...]  # from a 1.4MB json file
# about 30k such groups, results in ~70k replacements
count = 0
_col = df[column].astype("int64")
replace_dict = {}
for replace_list in groups:
        count +=1
        replacement = sorted(replace_list)[0]
        for replace_id in replace_list:
            if replace_id == replacement:
                continue
            replace_dict[replace_id] = replacement
        if count > 2000:
            _col = _col.replace(replace_dict)
            replace_dict = {}
            count = 0
_col = _col.replace(replace_dict) # for the last chunk

This works (takes about 11min) and 4.7GB of RAM are in use.

pandas._np_version
Out[3]: '1.8.0'

pandas.__version__
Out[4]: '0.13.1-489-g7ffa655'

I don't remember the numpy/pandas version from 10 month ago (The replacement file was generated on 17.05.2013, so during that week it worked) :-( It was probably whatever was on pandas master and the latest numpy release (I compile pandas from head and install numpy from cgolhlkes installers).

[jreback]

@cpcloud ?

[dsm054]

Something seems strange here.

@JanSchulz: how long does _col.apply(lambda x: replace_dict.get(x, x)) take?

[jankatins]

WTF:

replacements_by_XXX.json (16:35:42):  groups: 31612 replacements: 77846 =>  1.36s
replacements_by_XXXY.json (16:35:43):  groups: 19576 replacements: 33845 =>  1.03s
replacements_manual.json (16:35:44):  groups: 16 replacements: 17 =>  864ms
Wall time: 3.26 s

This took 20min with the above "2k chunks" code.

[jreback]

@JanSchulz I don't understand your problem exactly, but instead of replacing, why wouldn't you use these as indexers.

e.g. say you are replacing string to numbers; just construct a list (or series) of the strings, then
do s.take(numbers) (or s.iloc)?

will be orders of magnitude faster

[dsm054]

@JanSchulz: are those the numbers you get when you use the apply/get combination?

@jreback: I think the issue is that the following doesn't feel right at all:

>>> to_replace = {i: 10**7+i for i in range(10**6)}
>>> series_length = 10**3
>>> col = pd.Series(np.random.randint(0, 10**6, series_length))
>>> 
>>> %time slow = col.replace(to_replace)
CPU times: user 30 s, sys: 0 ns, total: 30 s
Wall time: 30 s
>>> %time fast = col.apply(lambda x: to_replace.get(x,x))
CPU times: user 4 ms, sys: 0 ns, total: 4 ms
Wall time: 1.53 ms
>>> 
>>> (slow == fast).all()
True

We should be getting O(n) values * O(1) lookup, making this very fast, and we're not. [Okay, I haven't varied the parameters enough to pick up the actual scaling -- could just be a really big constant, from what I've shown.]

[jreback]

hmm.....seems doing lots of work inside....

if you do this:

df = DataFrame({ 0 : col })
df.replace({ 0 : to_replace })

much faster....

prob a bug

[dsm054]

I don't actually trust {0: to_replace} at the moment; see the discussion in #6689.

[cpcloud]

this is very weird ... looking now

[cpcloud]

prun of @dsm054's example shows that BlockManager.replace_list is to blame...figuring out the line ... lprun 1.0b3 breaks with newest ipython 😢