thread safety

[stevengj]

If Julia's Threads.nthreads() > 1, we may want to do some more work to ensure thread safety. In particular:

• Call PyEval_InitThreads() in __init__ (this is only needed for Python ≤ 3.6, and is a no-op in Python ≥ 3.7).
• Acquire the GIL in the PyObject finalizer, by calling PyGILState_Ensure (returns an enum, i.e. a Cint in Julia) and PyGILState_Release. This is because the Julia GC may be called from threads other than the main thread, so we need to ensure that we hold Python's GIL before decref-ing.

We might also expose an API to acquire the GIL, but in general I would recommend that user code should only call Python from the main thread.

See this forum post and #881 for a possible test case.

[marius311]

Thanks, yea this sounds like it could be the cause. I tried your suggestion and it deadlocks. My guess is you have to do something like here. Playing with it, although I'm not an expert.

[stevengj]

I hope we don't have to do something as complicated as this code in FFTW.jl.

See the discussion in JuliaMath/FFTW.jl#141 where we ran into deadlocks during GC because FFTW was trying to acquire a mutex lock during plan destruction.

[stevengj]

Hmm, unfortunately I think we may be in the same situation as FFTW, because Python can call back to Julia. That is, the following may occur:

1. We call a Python function, which acquires the GIL.
2. Python calls back to Julia.
3. During the Julia call, Julia decides to run GC, which tries to finalize some PyObject
4. The PyObject finalizer tries to acquire the GIL. Deadlock ensues.

So, we may need something like the FFTW.jl solution after all.

[marius311]

Thanks, need to think about this and read that code, but one thing I dont totally follow is that in solution in my PR, the PyObject finalizer never tries to acquire the GIL. So are we fine?

[PallHaraldsson]

For some uses PythonCall.jl [EDIT: renamed from Python.jl] may be preferable? I'm not yet clear about the pros and cons, or why that new package needed, but at least it has some GIL-related code, so you could either use that package or maybe it's helpful to look at the code to fix here:

https://github.com/cjdoris/PythonCall.jl/blob/6bf7575247b0354009ae8f97387817f60bf45442/src/gil.jl

[secondaryfox]

I'm trying to run pytorch code from julia's pycall. I know that the pytorch code does not require GIL so I would like to run multiple of this code with julia threads -> pycall -> without GIL. Is this automatically achieved or should I do anything extra?

[mirkobunse]

Users of PyCall can use a global lock that every call to Python has to acquire.

module MyModule

const PYLOCK = Ref{ReentrantLock}()

function __init__() # instantiate the lock
    PYLOCK[] = ReentrantLock()
end

# acquire the lock before any code calls Python
pylock(f::Function) = Base.lock(f, PYLOCK[])

function example_function()
    pylock() do
       # any code that calls Python
    end
end

end # module

This approach works at least if Julia calls Python without any call-back to Julia. I assume that it also works with a call-back, as long as this call-back does not call Python again.

---

EDIT

I have verified the above approach with the following example, where we intend to fill the large array a with random numbers from numpy.random.rand().

using MyModule, PyCall

a = zeros(10000)
np_random = pyimport("numpy.random")

The following code, which does not acquire a lock, segfaults:

Threads.@threads for i in 1:length(a)
    a[i] = np_random.rand()
end

However, we can acquire the lock to achieve the desired effect.

Threads.@threads for i in 1:length(a)
    MyModule.pylock() do
        a[i] = np_random.rand()
    end
end

Of course, this locking mechanism requires all Python calls to acquire the lock. And acquiring the lock limits the degree to which our program parallelizes. In the above example, there is indeed no parallelism at all because the lock has to be acquired for each, complete iteration of the loop.