BUG: df1.values is df1_shallow_copy.values returns false

[TommasoBendinelli]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import pandas as pd
r = pd.DataFrame({"a": [1,2,3], "b": [3,4,5]})
shallow = r.copy(deep=False)
r.values is shallow.values

Problem description

I am making a shallow copy, hence I expect r and shallow's values to point to the same underlying data (as it is described here:https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.equals.html). Instead it is returning False

Expected Output

r.values is shallow.values should return True.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2a7d332
python : 3.7.6.final.0
python-bits : 64
OS : Darwin
OS-release : 18.7.0
Version : Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8

pandas : 1.1.2
numpy : 1.18.1
pytz : 2019.1
dateutil : 2.7.5
pip : 19.3.1
setuptools : 49.3.1
Cython : 0.29.14
pytest : 6.0.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.10.1
IPython : 7.6.0
pandas_datareader: None
bs4 : 4.8.2
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.0.2
numexpr : 2.7.1
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.2.0
sqlalchemy : None
tables : None
tabulate : 0.8.7
xarray : None
xlrd : None
xlwt : None
numba : None

[dsaxton]

My guess is values is actually copying under the hood (even df.values is df.values returns False). If you dig into the underlying blocks then you can "prove" the data wasn't copied:

import pandas as pd


print(pd.__version__)
# 1.1.2

df = pd.DataFrame({"a": [1,2,3]})
shallow = df.copy(deep=False)

print(df.values is shallow.values)
# False
print(df.values is df.values)
# False
print(df._mgr.blocks[0].values is shallow._mgr.blocks[0].values)
# True

[TommasoBendinelli]

Yes, indeed, the data it is not copied (if I assign an element to a data frame also the other is affected). It is just weird that to get this counter intuitive result

[jreback]

this might be ok after: #34872

but generally this is almost impossible to guarantee as once these input data is copied it is later combined to a new block which might or might not be viewable. I would not rely on this behavior at all.

If you pass in a 2-D numpy array then the semantics are more, but not perfectly clear.

[jorisvandenbossche]

Note that your example could also only ever work when you have a single dtype (and with the current internals of using consolidated blocks), once you have columns with different dtypes, .values will always be a copy.

The reason that data is a view but not identical, is because the data is stored under the hood in a transposed way. So basically when accessing .values, the stored array gets transposed:

In [15]: arr = np.array([[1, 2, 3], [4, 5, 6]]) 

In [16]: arr1 = arr.transpose()  

In [17]: arr2 = arr.transpose()  

In [18]: arr1 is arr2
Out[18]: False

So even when the underlying arr might not be copied, the returned value will each time be a new object viewing the same data.

[jorisvandenbossche]

@jreback note that the original example on top was about copying a dataframe, not about copying (or not) a numpy array passed to the DataFrame constructor.