BUG: DataFrame([listlikes]) different shape if first listlike is Categorical

[jbrockmendel]

arrs =  [np.arange(0, 40, 10), np.arange(0, 40, 10), np.arange(0, 40, 10)]

arrs2 = arrs.copy()
arrs2[0] = pd.Categorical(arrs2[0])

df = pd.DataFrame(arrs)
df2 = pd.DataFrame(arrs2)
df3 = pd.DataFrame(arrs2[::-1])

>>> df.shape
(3, 4)
>>> df2.shape
(4, 3)
>>> df3.shape
(3, 4)

It's specific to Categorical, as we check isinstance(data[0], Categorical) in nested_data_to_arrays

[venaturum]

Looks like the root cause is to_arrays, which is called by nested_data_to_arrays

(pandas/core/internals/construction.py)

def to_arrays(data, columns, dtype: Optional[DtypeObj] = None):
    """
    Return list of arrays, columns.
    """
    .
    .
    .
    elif isinstance(data[0], Categorical):
        if columns is None:
            columns = ibase.default_index(len(data))
        return data, columns

[venaturum]

There's a few functions in construction.py which seem to base behaviour around the first element's type in a nested list.

eg

pd.DataFrame([
    [1,2,3],
    pd.Series([4,5,6]),
    [7,8,9]
])

works but:

pd.DataFrame([
    pd.Series([1,2,3]),
    [4,5,6],
    [7,8,9]
])

results in an exception.

This begs the question, "what is the desired result when every nested array is a Categorical:

pd.DataFrame([pd.Categorical(arrs[i]) for i in range(3)])

Is it (by design):

	0	1	2
0	0	0	0
1	10	10	10
2	20	20	20
3	30	30	30

If it is, then this what is the desired result if all, but one, of the nested arrays are Categorical?

Seems like there needs to be clarification around design choices here. In any case, the order of the nested arrays shouldn't affect the final shape of the DataFrame and so using assumptions based on the type of the first nested array are not ideal.

[venaturum]

Is the solution as simple as replacing code in construction.py of the form:

isinstance(data[0], type_to_check)

with

all(map(lambda x: isinstance(x, type_to_check), data))

?

[jbrockmendel]

The first step is to get consensus that we want to change the behavior in the Categorical case. It may end up needing a deprecation cycle.

[jreback]

in an ideal world i think we would want to change this (though needing a deprecation cycle). @venaturum can you make the change locally and run asvs and see if performance is an issue (it might be)?

[venaturum]

in an ideal world i think we would want to change this (though needing a deprecation cycle). @venaturum can you make the change locally and run asvs and see if performance is an issue (it might be)?

Sure thing.

[venaturum]

@jreback

I've ran the asv. Do I zip and upload pandas/asv_bench/results?

What about the output in the terminal?

[jreback]

just post if anything is significantly different