Serializing Pandas Functions

[mrocklin]

In recent efforts using Pandas on multiple machines I've found that some of the functions are tricky to serialize. Apparently this might be due to runtime generation. Here are a few examples of serialization breaking, occasionally in unpleasant ways:

In [1]: import pandas as pd
In [2]: import pickle
In [3]: pd.read_csv
Out[3]: <function pandas.io.parsers._make_parser_function.<locals>.parser_f>
In [4]: pickle.loads(pickle.dumps(pd.read_csv))
AttributeError: Can't pickle local object '_make_parser_function.<locals>.parser_f'

Lest you think that this is just a problem with pickle (which has many flaws), dill, a much more robust function serialization library, also fails (the failure here is py35 only.) (cc @mmckerns)

In [5]: import dill
In [6]: dill.loads(dill.dumps(pd.read_csv))
PicklingError: Can't pickle <function _make_parser_function.<locals>.parser_f at 0x7f71f5ec1158>: it's not found as pandas.io.parsers._make_parser_function.<locals>.parser_f

In this particular case though cloudpickle will work.

Other functions have this problem as well. Consider the series methods:

In [7]: pickle.loads(pickle.dumps(pd.Series.sum))
AttributeError: Can't pickle local object '_make_stat_function.<locals>.stat_func'

In this case, concerningly cloudpickle completes, but returns a wrong result:

In [9]: import cloudpickle
In [11]: pd.Series.sum
Out[11]: <function pandas.core.generic._make_stat_function.<locals>.stat_func>

In [12]: cloudpickle.loads(cloudpickle.dumps(pd.Series.sum))
Out[12]: <function stat_func>

I've been able to fix some of these in cloudpipe/cloudpickle#46 but generally speaking I'm running into a number of problems here. It would be useful if, during the generation of these functions we could at least pay attention to assigning metadata like __name__ correctly. This one in particular confused me for a while:

In [15]: pd.Series.cumsum.__name__
Out[15]: 'sum'

What would help?

• Testing that most of the API is serializable
• Looking at what metadata the serialization libraries use, and making sure that this metadata is enough to properly identify the function. Some relevant snippets from cloudpickle follow:

    def save_instancemethod(self, obj):
        # Memoization rarely is ever useful due to python bounding
        if obj.__self__ is None:
            self.save_reduce(getattr, (obj.im_class, obj.__name__))
        else:
            if PY3:
                self.save_reduce(types.MethodType, (obj.__func__, obj.__self__), obj=obj)
            else:
                self.save_reduce(types.MethodType, (obj.__func__, obj.__self__, obj.__self__.__class__),
                         obj=obj)

    def _reduce_method_descriptor(obj):
        return (getattr, (obj.__objclass__, obj.__name__))

[max-sixty]

Why do you need to serialize the functions? If pandas is on the other machine, that shouldn't be necessary?

[mrocklin]

I need to communicate to a process on the other machine which pandas function to run.

[mmckerns]

@mrocklin: I'm not seeing dill have issue with pd.Series.sum. It would be nice to have a list of the serialization issues you are seeing with pandas. (Thanks for the pointer on the breakage for pandas with dill in python 3.5.)

Some adjustments to dill could be made to help you here, however, the answer is typically changes to the package (pandas) is the better answer. It's common that very little changes make all the difference, and usually have negligible negative impact on speed and functionality of the package. pandas is a fundamental enough package that I'd be willing to digest some special cases.

[jreback]

So the problem with the cum functions is trivial. Just naming them differently. So a PR for that is sort of easy.

diff --git a/pandas/core/generic.py b/pandas/core/generic.py
index 958571f..9764ae2 100644
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -4701,7 +4701,7 @@ class NDFrame(PandasObject):
             lambda y, axis: np.minimum.accumulate(y, axis),
             np.inf, np.nan)
         cls.cumsum = _make_cum_function(
-            'sum', name, name2, axis_descr,
+            'cumsum', name, name2, axis_descr,
             "cumulative sum",
             lambda y, axis: y.cumsum(axis), 0., np.nan)
         cls.cumprod = _make_cum_function(

how is the cloudpickle result for Series.sum wrong? this is just a closure

[mrocklin]

In [1]: import pandas as pd

In [2]: import cloudpickle

In [3]: s = pd.Series([1, 2, 3])

In [4]: sum2 = cloudpickle.loads(cloudpickle.dumps(pd.Series.sum))

In [5]: pd.Series.sum(s)
Out[5]: 6

In [6]: sum2(s)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-49effaf88fb4> in <module>()
----> 1 sum2(s)

TypeError: unbound method stat_func() must be called with NoneType instance as first argument (got Series instance instead)

This I think was more an error with cloudpickle than with pandas. This was fixed in a recent cloudpickle PR.

[jreback]

So in your example above its serializing an unbound method.

In [5]: sum2.__self__

You can serialize a bound method then it works

In [14]: sum2 = cloudpickle.loads(cloudpickle.dumps(s.sum))

In [15]: sum2
Out[15]: 
<bound method Series.stat_func of 0    1
1    2
2    3
dtype: int64>

In [16]: sum2()
Out[16]: 6

[mrocklin]

In this case I want to serialize an unbound method.

[jreback]

so this is a cloudpickle/dill issue then? (aside from the naming in the cum* funcs)?

[mrocklin]

It can be handled in either. Like Mike said earlier, a little bit of help from Pandas can go a long way here. It's important to make sure that all functions maintain metadata like __name__ and __self__ and im_class through all of the manipulations that Pandas does.