DataFrame.copy(), at least, should be threadsafe

[bshanks]

dataframe.copy() should happen atomically/be threadsafe, meaning that it should produce a consistent dataframe even if the call to .copy() is made while another thread is deleting entries from the dataframe, or if another thread calls a deletion method while the call to .copy() is working (in other words, i guess .copy() should acquire a lock that prevents mutation during the copy). That is, the following code, which crashes in 0.7.3, should succeed:

import pandas
import threading

df = pandas.DataFrame()

def mutateDf(df):
    while True:
        df[0] = pandas.Series([1,2,3])
        del df[0]

def readDf(df):
    while True:
        dfCopy = df.copy()
        if 0 in dfCopy and 1 in dfCopy[0]:
            a = dfCopy[0][1]

t1 = threading.Thread(target=mutateDf, args=(df,))
t2 = threading.Thread(target=readDf, args=(df,))

t1.start()
t2.start()

Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "<ipython-input-5-8aef72c7f1b4>", line 4, in readDf
    if 0 in dfCopy and 1 in dfCopy[0]:
  File "/usr/local/lib/python2.7/dist-packages/pandas-0.7.3-py2.7-linux-x86_64.egg/pandas/core/frame.py", line 1458, in __getitem__
    return self._get_item_cache(key)
  File "/usr/local/lib/python2.7/dist-packages/pandas-0.7.3-py2.7-linux-x86_64.egg/pandas/core/generic.py", line 294, in _get_item_cache
    values = self._data.get(item)
  File "/usr/local/lib/python2.7/dist-packages/pandas-0.7.3-py2.7-linux-x86_64.egg/pandas/core/internals.py", line 625, in get
    _, block = self._find_block(item)
TypeError: 'NoneType' object is not iterable

Right now, pandas is explicitly not thread-safe. Taking any step down this path
will inevitably generate lots of pain and changes all over. Python threads see more
limited use then in other languages, the upside is correspondingly limited.

You can always implement per-object or a global pandas lock in your own code,
if threads are what you want.

Pushing back to 0.12, at least.

[wesm]

That's not quite true-- for example most things are threadsafe and we've ensured that e.g. IO functions can be run in separate threads. Perhaps we should just acquire a lock inside the copy functions for now

[jreback]

copy might be thread safe with a single dtype (but prob not)
multiple dtypes now are not thread safe
(as @wes points out a lock will fix all this)
I would be in favor of providing this as an option, default to False though

I was thinking of #2440. Perhaps parts of pandas are thread-safe, but
afik there's no list of what's safe or not and users have hit the non-safe
parts before this, when they tried.

[kokes]

I cannot replicate the error posted, see this gist. Either there have been new developments in atomicity of pandas or perhaps threading has a different scheduler, or...?

(Tried under both Python 2.7 and 3.5)

[jreback]

this almost certainly had to do with the unsafe-threadness (is that a word?) with numexpr. numexpr>=2.5 (and even >=2.4.6) now don't much with the global thread state. @kokes what version do you have?

[kokes]

Good! I've got 2.4.6 (under conda), upgraded to 2.5 and got the same.

[allComputableThings]

copy might be thread safe with a single dtype

Doubtful: See: #25870

It seems, you can't can't currently use pandas series for a 'read-only' hash-lookup in a threaded environment.

It fails on the second call to Series.reindex(..., copy=True) - I was extremely surprised by it, thinking the operation to be non-mutating. I would have expected any hidden object state, such as built indexes, to be finalized at the end of the first call, and subsequent calls to be safe.