iloc breaks on read-only dataframe

[amueller]

This is picking up #9928 again. I don't know if the behavior is expected, but it is a bit odd to me. Maybe I'm doing something wrong, I'm not that familiar with the pandas internals.

We call df.iloc[indices] and that breaks with a read-only dataframe. I feel that it shouldn't though, as it is not writing.

Minimal reproducing example:

import pandas as pd
import numpy as np
array = np.eye(10)
array.setflags(write=False)

X = pd.DataFrame(array)
X.iloc[[1, 2, 3]]

ValueError buffer source array is read-only

Is there a way to slice the rows of the dataframe in another way that doesn't need a writeable array?

[amueller]

Maybe to clarify: I would be happy if the sliced DF is a copy, I just don't want to copy the original data [I think].

[jreback]

by definition what you are doing is ALWAYS going to be a copy. Frames are stored in columns.

In [10]: df = DataFrame(np.arange(20).reshape(5,4))

In [11]: df
Out[11]: 
    0   1   2   3
0   0   1   2   3
1   4   5   6   7
2   8   9  10  11
3  12  13  14  15
4  16  17  18  19

In [12]: result = df.iloc[[1,2,3]]

In [13]: result
Out[13]: 
    0   1   2   3
1   4   5   6   7
2   8   9  10  11
3  12  13  14  15

In [14]: result.values.base
Out[14]: 
array([[ 4,  5,  6,  7],
       [ 8,  9, 10, 11],
       [12, 13, 14, 15]])

In [15]: df._data
Out[15]: 
BlockManager
Items: Int64Index([0, 1, 2, 3], dtype='int64')
Axis 1: Int64Index([0, 1, 2, 3, 4], dtype='int64')
IntBlock: slice(0, 4, 1), 4 x 5, dtype: int64


In [17]: df._data.blocks[0].values
Out[17]: 
array([[ 0,  4,  8, 12, 16],
       [ 1,  5,  9, 13, 17],
       [ 2,  6, 10, 14, 18],
       [ 3,  7, 11, 15, 19]])

you can just access using [17] if you want to get the actual numpy array. Keep in mind, this is ONLY valid for a single-dtyped frame.

[TomAugspurger]

Would it be OK for scikit-learn to rely on ._data / blocks in library code?

This also isn't great since I've passed DataFrames into scikit-learn that are mixes of int / float. I'm guessing that's a pretty common thing.

[amueller]

Getting a copy of the slice if fine, I'd just like to avoid copying the whole initial data.
I would like to retain the dataframe structure.

Can you elaborate what it means that this is only valid for single-dtyped frames?

I would like to have something that works on mixed type and single dtype frames and produces a dataframe.

To elaborate on the use-case:
We have a function that does some slicing, cross_val_score that takes a model. I want the slicing to work and not destroy the dataframe, in case the user provided a custom model that wants a dataframe.

[jreback]

then use use .iloc as you have indicated. I am not sure why you'd have a read-only array in the first place. I suppose that could be handled (it actually borks because its taking with a memory view).

Slicing doesn't touch the original data at all. It will make a copy. (It will hand you a view if its possible, but this is cross-sectional slicing so that is numpy dependent, but ONLY if you have a single-dtype). If you have multiple dtypes, you will ALWAYS get a copy.

What is the actual error you are getting?

[shoyer]

External libraries should not be using the private pandas block system _data. I think that suggestion is a non-starter.

I agree that it is weird that read-only arrays don't work with .iloc. The ultimate problem appears to be that Cython memory views can't be created on read-only arrays:
https://mail.python.org/pipermail/cython-devel/2013-February/003384.html

To fix this on our end, we should probably wrap our calls to the internal take_nd Cython routines with something that ensures the source arrays are writable. Given that we aren't actually writing to them, I think that should be safe.

[amueller]

The traceback is in #9928, I agree with @shoyer that it seems to be cause by what is described in the cython thread. The solution sounds ok to me but as I said I don't know much about the internals of pandas.

The non-writeable array comes from joblib. I am not entirely sure about the background and I will investigate if we can get rid of it. I was not aware of the cython behavior when I opened the issue and hoped I was actually doing something wrong.

[shoyer]

Ultimately, I think this issue should be solved in on the Cython end of things -- there are plenty of legitimate users for readonly memory views. In fact @richardhansen reports in this this stackoverflow answer that he has a rough patch for this.

In the meantime, does anyone know how to create a "writeable" array from a readonly array with (a) not making any copies of the underlying data and (b) no modifications to the source array? This does not seem to work:

array = np.arange(10.0)
array.setflags(write=False)

array2 = np.asarray(array)
array2.setflags(write=True)
print array.flags.writeable

True

[amueller]

Ok, I think it is fair of you to punt this to cython. I have to work around it in sklearn anyhow for the moment (even if you fixed it we want to support current stable pandas etc).
I think asarray doesn't do anything if you pass it an array, but

array = np.arange(10.0)
array.setflags(write=False)

array2 = np.array(array)
array2.setflags(write=True)
print array.flags.writeable

False

[amueller]

Feel free to close.

[shoyer]

@amueller The problem with calling np.array is that you're copying the data... which is generally not ideal (especially for a routine like .iloc, which is supposed to be fast). (I just updated my comment from 8 minutes ago to clarify what I'm looking for.)

[amueller]

brainfart. With copy=False it also gives "True".