ValueError: EXT data is too large

[randomgambit]

Hi guys,

I am happy to help you improve msgpack.
I tried to export my massive dataframe this morning using msgpack, and I got this error.

ValueError: EXT data is too large

What does that mean? Is there a size limit?

[jreback]

@randomgambit if you can post something more.

cc @kawochen

[randomgambit]

hi jeff i dont have my computer in front of me. but its the same exact dataframe as my post on the slowness of to_csv

[randomgambit]

tell me what information you need

[jreback]

ahh ok

can u reference that issue here as well then (and post the df.info()

it may break if u have an object column that actually has object types in it (and not strings)

[randomgambit]

hello @jreback @kawochen

df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 10762587 entries, 0 to 12864511
Columns: 275 entries, bagent_name to index_month
dtypes: bool(1), datetime64[ns](16), float64(30), int32(1), int64(172), object(53), timedelta64[ns](2)
memory usage: 22.0+ GB

other problems related to this df are discussed here #12885

hope that helps

[jreback]

does this work with a smaller slice of your frame?

Do you have any non-string object data? IOW, run something like

In [1]: df = tm.makeMixedDataFrame()

In [2]: df
Out[2]: 
     A    B     C          D
0  0.0  0.0  foo1 2009-01-01
1  1.0  1.0  foo2 2009-01-02
2  2.0  0.0  foo3 2009-01-05
3  3.0  1.0  foo4 2009-01-06
4  4.0  0.0  foo5 2009-01-07

In [3]: df.apply(pd.lib.infer_dtype)
Out[3]: 
A    floating
B    floating
C      string
D    datetime
dtype: object

If you get anything like mixed-.... then you need to stringfy (or coerce) before this would be expected to work.

storing giant opaque frames like this is generally not very useful as it forces you to load them entirely into memory to work with them.

[randomgambit]

Hi Jeff aka @jreback

Thanks! A couple of points

• most of my work consists in doing regression analysis on large data samples. That means its actually a necessity for me to load the whole sample in memory (and to have a quick n dirty way to save and load my data at any step of the processing step)
• I used your df.apply(pd.lib.infer_dtype) and checked for mixed types. BTW THAT is a great function that I did not know about. I recommend you put that in the tutorial (under "do I really know my data types?" ;-)
• once I ran to_msg it seemed to work, going from 0 to 2 GB on my disk after 30 min of processing. Then it stayed there for a long time so I killed it. I dont know if msgpackis supposed to write continuously (so hitting F5 like crazy actually shows the msg file growing) or is doing some sort of chunk-by-chuck exporting. I ll try asap with a smaller sample

[randomgambit]

@jreback is it possible to write to msgpackby iterating over the dataframe? I suspect trying to export the whole dataframe at once is too heavy a task

[jreback]

it's possible but not efficient
I think this is because of an unknown datatype

pls see if u can narrow down exactly when his occurs (eg remove columns until it works)

[kawochen]

this line . It's not clear to me why it's testing for (2**32)-1. i think we can try (size_t) -1 instead.

[jreback]

I wonder if msgpack uses a single byte for size which is limited

[kawochen]

xref msgpack/msgpack-python#181

[kawochen]

In the C code size_t is used.