Make itertuples really an iterator/generator in implementation, not just return type

[mitar]

itertuples is not really an iterator/generator and constructs a copy of whole DataFrame in memory. Ideally it would return just an iterator and construct row by row as it is being iterated over.

[jreback]

looks like a generator to me

In [1]: df = pd.DataFrame({'A': range(3), 'B': list('ABC')})

In [2]: df.itertuples()
Out[2]: <map at 0x10922c080>

In [3]: list(df.itertuples())
Out[3]: 
[Pandas(Index=0, A=0, B='A'),
 Pandas(Index=1, A=1, B='B'),
 Pandas(Index=2, A=2, B='C')]

In [5]: i = df.itertuples()

In [6]: next(i)
Out[6]: Pandas(Index=0, A=0, B='A')

In [7]: next(i)
Out[7]: Pandas(Index=1, A=1, B='B')

In [8]: next(i)
Out[8]: Pandas(Index=2, A=2, B='C')

In [9]: next(i)
---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
<ipython-input-9-a883b34d6d8a> in <module>()
----> 1 next(i)

StopIteration: 

[mitar]

That's just because you return a zip. But inside the function, you create whole list of values first. See here:

arrays.extend(self.iloc[:, k] for k in range(len(self.columns)))

It looks like iterator because of zip(*arrays), but arrays is not an iterator. This is a problem.

[mitar]

You can test the issue here by doing:

d = pandas.DataFrame({'a': range(100000000)})
for a in d.itertuples(index=False, name=None):
    print(a)

Do this in Python interpreter. Note the time it takes to create d. Now, when you press enter after print line, twice, note the time it takes to start producing anything. It takes this time because it is first creating all rows in memory, before iterating over them.

[jreback]

and if u want to fix it pls submit a PR
this is truly an iterator

so what u describe is an implementation detail

[mitar]

Implementation detail which blows up memory and performance?

Anything can be made look like iterator. But if does not really behave like iterator, it is not an iterator.

I think this is a bug. Please reopen this. And then me or somebody else can make a pull request.

[mitar]

This goes even deeper. Also iterating over a series constructs a list internally:

d = pandas.DataFrame({'a': range(100000000)})
for a in d['a']:
    print(a)

This will also have a large delay before starting sending results back.

[jorisvandenbossche]

I agree that ideally it would be more lazy the iteration (although not really a priority issue for me), and since we would accept a PR to fix, let's keep the issue open.