Expose the blocks API and disable automatic consolidation

[shoyer]

In my discussion with Jonathan and others and at the SciPy sprints, we agreed that it would be really nice to expose some minimal tools for manipulating and view the internal pandas blocks system. For example, it should be possible to:

1. manually consolidate blocks
2. view a representation of the internal blocking of a dataframe (via matplotlib?)

It's not so much that we want to create and use blocks directly, but that we want to make it easier to understand the internal data model and make performance with more predictable.

At the same time, we would like to disable automatic consolidation of blocks in the DataFrame constructor and when inserting new columns. Consolidation is certainly a useful feature, but it is currently not always possible to even predict when it will happen.

Most users never notice or care about consolidation. Power users (concerned about memory or performance) are at least as likely to find it frustrating as helpful, so we should make this something that they can trigger explicitly (as part of the blocks API). This would make it possible to create dataframes while guaranteeing that none of the data is copied (#9216).

cc @jonathanrocher @sinhrks @jreback @cpcloud @TomAugspurger @ARF1 @quicknir

[jonathanrocher]

Yes, I am definitely a big +1 on both of these ideas. I need to understand the current state better, before experimenting with such a tool. Just to clarify what @shoyer means by the blocks API, we discussed exposing these view and consolidation methods via a blocks attribute: df.blocks.view(), df.blocks.consolidate(), or something like that.

[jreback]

well

df.as_blocks(copy=False) (copy flag added in master recently) gives you a view right now

and as discussed, I would like to see why this should actually be exposed to the user. What is the purpose? A good use case would go a long way. Showing an implementation is only guaranteed to cause issues down the road when one wants to change it (and cannot because its exposed)

[shoyer]

I don't think we should expose the full blocks API. Rather, we should expose a limited set of functionality suitable for basic understand and manipulation of blocks (mostly .consolidate_blocks()), so we can have the option of disabling automatic consolidation. If we ever change the underlying implementation, we can just make this method a no-op.

For viewing block information, we might expose a tuple, similarly to the .chunks attribute on dask arrays.

[sinhrks]

I assume exposing some API is for let power users experiment, rather than daily usage? I think adding more detail to internals.rst is very nice (#4082), but not sure what level of "public" is actually required (guarantee backward compat or adding to API.rst, etc)

+1 for optionally disabling auto consolidation, it should allow pandas handle larger size of data.

[jreback]

I still have not seen any reason to actually expose anything about the internals as an API.

It seems that one might want to control how the internal engine works, e.g. simply have an option on construction that allows potentially different internal representation.

No consolidation is a nice idea on the surface and their may be a reason to allow this and/or switch as a default. HOWEVER, I would like to see someone do a performance study to see if it actually makes a difference in a non-trivial case.

Turning off auto-consolidation is actually trivial, so if someone would like to step up, all ears.

[quicknir]

@shoyer Thanks for tagging me on this. I am a strong +1 on this. As an example of a use case, ARF1 showed in #9216 how one could use the block manager interface to create an empty DataFrame nearly 100 times faster than the naive approach. It would be nice if one could do this without accessing _implementation methods that are subject to change without warning.

When developing a C/C++ library that interfaces with pandas, memory layout is critical. Increasing public interface that allows better guarantees and more explicit behavior of memory layout is very helpful. I can give other examples in pandas where similar issues arise.

[jreback]

jreback@44c280b

is a branch that allows a parameter policy to control how consolidation works.

Its not completely automatic at this point (e.g. you have to pass policy='split|column' to get this behavior (which could be an option, support.consolidation). Further for a dict we can simply default to column I think.

note that:

• column -> don't consolidate, IOW, this is a 'column' repr internally. This won't 'split' a passed in ndarray though (e.g. its still a single block to begin with). This could eventually be the default.
• split -> force split a passed ndarray. This can be costly, but with the upside that you get proper memory layout. This has to be a user decision as its not always necessary / wanted.
• needs some more tests and such
• needs some perf testing
• need to have various ops (e.g. concat), deal with the policy properly (may be non-trivial)

[jreback]

Here are some perf considerations

In [3]: np.random.seed(1234)

# blocks
In [4]: df_blocks = DataFrame(np.random.randn(100000,1000))

# create a single ndarray, then split it
In [5]: df_split = DataFrame(np.random.randn(100000,1000),policy='split')

# create a dict of ndarrays, no need to split
In [10]: df_dsplit = DataFrame(dict([ (i,np.random.randn(100000)) for i in range(1000) ]),policy='split')

Creations

# block
In [7]: %timeit DataFrame(np.random.randn(100000,1000))
1 loops, best of 3: 3.79 s per loop

# ndarray-split
In [6]: %timeit DataFrame(np.random.randn(100000,1000),policy='split')
1 loops, best of 3: 5.93 s per loop

# dict-split
In [14]: %timeit DataFrame(dict([ (i,np.random.randn(100000)) for i in range(1000) ]),policy='split')
1 loops, best of 3: 3.93 s per loop

row-wise ops

# blocked
In [8]: %timeit df_blocks.sum()
1 loops, best of 3: 1.55 s per loop

# ndarray-split
In [9]: %timeit df_split.sum()
1 loops, best of 3: 607 ms per loop

# dict-split
In [15]: %timeit df_dsplit.sum()
1 loops, best of 3: 617 ms per loop

So you may be suprised by the speed of the splits. This is because these are already c-contiguous, while the block memory is NOT. Its in c-order, but not for a row-wise op.