PERF: Unnecessary hash table with RangeIndex

[chris-b1]

Example

def log_memory():
    import os
    import gc
    import psutil
    for i in range(3):
        gc.collect(i)
    process = psutil.Process(os.getpid())
    mem_usage = process.memory_info().rss / float(2 ** 20)
    print("[Memory usage] {:12.1f} MB".format(
        mem_usage
    ))

In [20]: df = pd.DataFrame({'a': np.arange(1000000)})

In [23]: log_memory()
[Memory usage]        132.4 MB

In [24]: df.loc[5, :]
Out[24]: 
a    5
Name: 5, dtype: int32

In [25]: log_memory()
[Memory usage]        172.2 MB

Rather than materializing the hash table, should directly convert labels into positions. Low priority in my opinion, atypical to be using loc with a RangeIndex.

pandas 0.20.2

[jreback]

hmm ,this should not be creating a hash table at all (though it would if it morphed to an int64index)

[chris-b1]

The _engine for RangeIndex still points at the Int64Index one, so a hash table will be created if something calls self._engine. So either need to make a simplified RangeIndexEngine or override the indexing methods.

pandas/pandas/core/indexes/range.py

Line 43 in 2e24a8f

_engine_type = libindex.Int64Engine

[jreback]

I think that _engine is purely for compat on some methods. might be easiest/best to create a pretty dummy RI Engine in cython.

[jorisvandenbossche]

I think that _engine is purely for compat on some methods

_engine is still used in eg get_loc, so often created.

Related to this, I just stumbled on the fact that currenlty RangeIndex.memory_usage() is inaccurate, as it does not take into account the engine.

And given that the actual engine takes a lot more memory as the values (for int64index it is 5x more memory on an example case 500k range-like int index), a RangeIndex is not much more memory-efficient as a Int64Index (its big selling point I thought?) once the engine is created.

[chris-b1]

RangeIndex is not much more memory-efficient as a Int64Index (its big selling point I thought?) once the engine is created.

This is true, but I think it would be relatively rare for the engine to populated when a RangeIndex is being used - typically it is when someone doesn't care about the index at all, so would likely be using iloc if any indexing at all.