BUG: iloc setting columns not taking effect

[jbrockmendel]

arr = np.random.randn(10 ** 6).reshape(500, 2000).astype(np.float64)
df = pd.DataFrame(arr)
df.iloc[:, 1000:] = df.iloc[:, 1000:].astype(np.float32)

>>> df.dtypes.value_counts()
float64    2000
dtype: int64

I would expect 1000 float64 columns and 1000 float32 columns. I get the expected behavior if I set a single column before trying to set all 1000:

df = pd.DataFrame(arr)
df[1000] = df[1000].astype(np.float32)
df.iloc[:, 1000:] = df.iloc[:, 1000:].astype(np.float32)

>>> df.dtypes.value_counts()
float32    1000
float64    1000
dtype: int64

[jbrockmendel]

@MomIsBestFriend this is non-trivial, might be worth taking a shot at

[ShaharNaveh]

@MomIsBestFriend this is non-trivial, might be worth taking a shot at

@jbrockmendel I Will definitely give it a shot, thank you for giving me things I might be able to work on!

[ShaharNaveh]

@jbrockmendel I have tried to solve this, but I am no where near close to do this, I guess this one is non-trivial 😞

But I did learn a lot about pdb while trying to solve this, so I guess that wasn't all a waste of time :)

[jbrockmendel]

how about let's walk through the process [i would use] to address this?

The line that isn't working as expected is df.iloc[:, 1000:] = df.iloc[:, 1000:].astype(np.float32). What is the actual function call that this is doing?

[ShaharNaveh]

how about let's walk through the process [i would use] to address this?

The line that isn't working as expected is df.iloc[:, 1000:] = df.iloc[:, 1000:].astype(np.float32). What is the actual function call that this is doing?

This takes a slice of df and cast it to be with dtype of float32, and then assigning it back into df into the position of the taken slice?

---

Edit:
this call __setitem__

[jbrockmendel]

yes. more specifically, this is equivalent to

key = (slice(None), slice(1000, None))
value = df.iloc[:, 1000:].astype(np.float32)

df.iloc.__setitem__(key, value)

You can check that value here has the expected dtypes. So lets look at the __setitem__ call. Where inside that call should we look at first?

[ShaharNaveh]

I have looked mainly at

pandas/pandas/core/indexing.py

Lines 657 to 666 in 70086c5

	def __setitem__(self, key, value):
	if isinstance(key, tuple):
	key = tuple(com.apply_if_callable(x, self.obj) for x in key)
	else:
	key = com.apply_if_callable(key, self.obj)
	indexer = self._get_setitem_indexer(key)
	self._has_valid_setitem_indexer(key)
	iloc = self if self.name == "iloc" else self.obj.iloc
	iloc._setitem_with_indexer(indexer, value)

[ShaharNaveh]

I have looked mainly at

pandas/pandas/core/indexing.py

Lines 657 to 666 in 70086c5

	def __setitem__(self, key, value):
	if isinstance(key, tuple):
	key = tuple(com.apply_if_callable(x, self.obj) for x in key)
	else:
	key = com.apply_if_callable(key, self.obj)
	indexer = self._get_setitem_indexer(key)
	self._has_valid_setitem_indexer(key)
	iloc = self if self.name == "iloc" else self.obj.iloc
	iloc._setitem_with_indexer(indexer, value)

Tried to have things like:

 key = com.apply_if_callable(key, self.obj, dtype=self.obj.dtypes.dtype) 

[jbrockmendel]

(stop me if you dont want to engage in this exercise. i think it may be useful)

thats the right place to look. So setting self = df.iloc and walking through this, 658 is True, 659 is going to end up returning key unchanged, and 662 is going to end up giving indexer = key. 665 gives iloc = self, so let's look at _setitem_with_indexer

Looking at _setitem_with_indexer, where do things start to go wrong?

[ShaharNaveh]

(stop me if you dont want to engage in this exercise. i think it may be useful)

It's very useful!

thats the right place to look. So setting self = df.iloc and walking through this, 658 is True, 659 is going to end up returning key unchanged, and 662 is going to end up giving indexer = key. 665 gives iloc = self, so let's look at _setitem_with_indexer

Looking at _setitem_with_indexer, where do things start to go wrong?

I don't know, can you elaborate?

[jbrockmendel]

So looking through _setitem_with_indexer, from 1549-1574 its not doing any setting, just figuring out take_split_path. What is take_split_path going to end up being?

[ShaharNaveh]

So looking through _setitem_with_indexer, from 1549-1574 its not doing any setting, just figuring out take_split_path. What is take_split_path going to end up being?

False

[ShaharNaveh]

@jbrockmendel removing the not at L1562 solves this issue!
I guess that in order to not break other things, there needs to be some extra logic.