Confusing behaviour of df.empty

[phil20686]

This is as much a documentation issue as anything else. Basically it seems confusing that df.empty != df.dropna().empty I.e. that a a dataframe consiting entirely of na is not treated as empty. Obviously this is a bit of an edge case, but it caused a bunch of failures for me when used eith pd.read_sql methods, as database tables will often have columns that are not available for partiicular entities, and so can return an entire series of na.

It seems to me that in all cases df.empty should be the same as df.dropna().empty, but I understand that opinions might differ on this point, but at least the behaviour should be clearly documented.

[jreback]

Can you show an example, the following is works. I agree certainly could update the documentation (with some examples and such)

In [8]: df = DataFrame({'A' : [np.nan]})

In [9]: df.empty
Out[9]: False

In [10]: df.dropna().empty
Out[10]: True

[phil20686]

That is exactly the behavior I was questioning, I think out[9] should be True. It seems to me that a dataframe containing nothing by na cells is "empty" according to most definitions.....

Phil

[phil20686]

I certainly expected that df.empty would be true if a data frame contained nothing but na cells.

[jreback]

no, that is not a normal definition of empty which is 0-len. nulls are real values which are placeholders. The key here is that you actually have a valid index. Changing this would involve the definition dependent on the data itself which is not a good thing. welcome to have a doc update with some examples though.

[phil20686]

Well its certainly not the case that df.empty is the same as len(df.index==0) e.g.

df = pd.DataFrame([], index=[0,1,2])
print df.empty #True
print len(df.index==0) #False

Also

df = pd.DataFrame([], index=[0,1,2], columns=['A','B'])
df.empty #False

[phil20686]

So not only do I dispute that len=0 is the semantic definition of empty, that doesn't appear to be the implementation anyway.

[phil20686]

Also stuff like:

df = pd.DataFrame([], index=[0,1,2], columns=['A','B'])
df.set_value(1, 'A', 17)
df['B'].empty # False

which just seems plain wrong. If I have a column that I have never added any data to, it should not return false when asked if its empty. I guess under the hood when you specify an index and columns it autofills the dataframe somehow, and that results in this behaviour, but the definition of empty should really play nicely with the default dataframe constructor in these examples imo. Else its just confusing.

[jreback]

its very simple. its empty only if all axes are len 0

[kawochen]

It sounds like you're looking for some other collection of Series. A DataFrame is a tabular collection, and it makes sense to look at the shape.

[jorisvandenbossche]

@phil20686 You can eg use:

In [15]: df['B'].isnull().all()
Out[15]: True