Segmentation fault with numpy.fft.rfft and MKL and threads

[LukasDrude]

When using numpy.fft.rfft with mkl_fft in a ThreadPoolExecutor you can run in a segmentation fault.

Why is this a cross-post?
Since this issue is related to Anaconda, Numpy and mkl_fft, this issue is posted on all three locations. Currently, it is not quite clear which party should address the issue.

• mkl_fft: No response yet.
• numpy: Sees issue at mkl_fft but uses a seemingly buggy library (namely mkl_fft).
• anaconda: No response yet.

We get the same error when testing this on different infrastructure/ OS:

• Scientific Linux 7.2 (Nitrogen)
• Ubuntu 16.04.4 LTS
• OSX

import numpy as np
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor


def fun(_):
    print(_)
    frames = np.random.randint(1000, 2000)
    signal = np.ones((2, frames), dtype=np.float64)

    _ = np.fft.rfft(signal)

    return 1,


if __name__ == '__main__':
    # print('ProcessPoolExecutor')
    # with ProcessPoolExecutor(4) as ex:
    #     list(ex.map(fun, range(50)))

    print('ThreadPoolExecutor')
    with ThreadPoolExecutor(4) as ex:
        list(ex.map(fun, range(500)))

Repeating the example code many times yields some kind of traceback, if it did not segfault.

Sometimes occuring traceback

Traceback (most recent call last):
  File "test_script.py", line 35, in <module>
    list(ex.map(fun, range(50)))
  File "/net/home/pyadmin/conda/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator
    yield fs.pop().result()
  File "/net/home/pyadmin/conda/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/net/home/pyadmin/conda/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/net/home/pyadmin/conda/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "test_script.py", line 11, in fun
    _ = np.fft.rfft(signal)
  File "/net/home/pyadmin/conda/lib/python3.6/site-packages/mkl_fft/_numpy_fft.py", line 331, in rfft
    output = mkl_fft.rfft_numpy(a, n=n, axis=axis)
  File "mkl_fft/_pydfti.pyx", line 569, in mkl_fft._pydfti.rfft_numpy
  File "mkl_fft/_pydfti.pyx", line 487, in mkl_fft._pydfti._rc_fft1d_impl
ValueError: Internal error occurred

Not working environment:

Conda installed numpy version: Version: 1.14.3

Numpy config

python -c "import numpy; print(numpy.show_config())"
mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/Users/lukas/anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/Users/lukas/anaconda3/include']
blas_mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/Users/lukas/anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/Users/lukas/anaconda3/include']
blas_opt_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/Users/lukas/anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/Users/lukas/anaconda3/include']
lapack_mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/Users/lukas/anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/Users/lukas/anaconda3/include']
lapack_opt_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/Users/lukas/anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/Users/lukas/anaconda3/include']
None

Working Environment:

Pip installed numpy version: 1.14.5

Numpy config

python -c "import numpy; print(numpy.show_config())"
blas_mkl_info:
  NOT AVAILABLE
blis_info:
  NOT AVAILABLE
openblas_info:
  NOT AVAILABLE
atlas_3_10_blas_threads_info:
  NOT AVAILABLE
atlas_3_10_blas_info:
  NOT AVAILABLE
atlas_blas_threads_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
blas_opt_info:
    extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
lapack_mkl_info:
  NOT AVAILABLE
openblas_lapack_info:
  NOT AVAILABLE
openblas_clapack_info:
  NOT AVAILABLE
atlas_3_10_threads_info:
  NOT AVAILABLE
atlas_3_10_info:
  NOT AVAILABLE
atlas_threads_info:
  NOT AVAILABLE
atlas_info:
  NOT AVAILABLE
lapack_opt_info:
    extra_compile_args = ['-msse3']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
None

[oleksandr-pavlyk]

Thank you for bringing the issue up and sorry you are running into issues with mkl_fft.

I was able to reproduce the problem with 1.0.2 as you described, seeing ValueError raised short ways into parallel execution:

  File "mkl_fft/_pydfti.pyx", line 491, in mkl_fft._pydfti._rc_fft1d_impl
ValueError: Internal error occurred

The value indicates that underlying call to MKL returned a non-OK status.

I was able to reproduce this behavior with NumPy 1.14.5 and 1.14.3 and mkl_fft 1.0.2. I can not reproduce the crash with the current master of mkl_fft though.

You can instruct NumPy in Anaconda, or Intel Distribution for Python to not use mkl_fft by executing

import numpy as np
np.fft.restore_all()

Hope this helps. I will report to this issue if I discover more.

[LukasDrude]

It is nice to see, that you can not reproduce it in MASTER. Then maybe you can help escalate it at the Anaconda site: ContinuumIO/anaconda-issues#9697

I was not sure if that Bug Tracker is the right place.

[oleksandr-pavlyk]

@LukasDrude The fix is availble on conda-forge channel right now. It will make its way into Anaconda main channel shortly.

[TAdeJong]

I experienced the same problem. I installed the new 1.0.3 version from conda-forge, but I am still experiencing the same problem, albeit with a slightly different error message:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-f117964a59d1> in <module>()
     20     print('ThreadPoolExecutor')
     21     with ThreadPoolExecutor(4) as ex:
---> 22         list(ex.map(fun, range(500)))

C:\ProgramData\Anaconda3\lib\concurrent\futures\_base.py in result_iterator()
    584                     # Careful not to keep a reference to the popped future
    585                     if timeout is None:
--> 586                         yield fs.pop().result()
    587                     else:
    588                         yield fs.pop().result(end_time - time.time())

C:\ProgramData\Anaconda3\lib\concurrent\futures\_base.py in result(self, timeout)
    423                 raise CancelledError()
    424             elif self._state == FINISHED:
--> 425                 return self.__get_result()
    426 
    427             self._condition.wait(timeout)

C:\ProgramData\Anaconda3\lib\concurrent\futures\_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

C:\ProgramData\Anaconda3\lib\concurrent\futures\thread.py in run(self)
     54 
     55         try:
---> 56             result = self.fn(*self.args, **self.kwargs)
     57         except BaseException as exc:
     58             self.future.set_exception(exc)

<ipython-input-1-f117964a59d1> in fun(_)
      8     signal = np.ones((2, frames), dtype=np.float64)
      9 
---> 10     _ = np.fft.rfft(signal)
     11 
     12     return 1,

C:\ProgramData\Anaconda3\lib\site-packages\mkl_fft\_numpy_fft.py in rfft(a, n, axis, norm)
    329         a = asarray(a)
    330         n = a.shape[axis]
--> 331     output = mkl_fft.rfft_numpy(a, n=n, axis=axis)
    332     if unitary:
    333         output *= 1 / sqrt(n)

mkl_fft\_pydfti.pyx in mkl_fft._pydfti.rfft_numpy()

mkl_fft\_pydfti.pyx in mkl_fft._pydfti._rc_fft1d_impl()

ValueError: Internal error occurred, with status=-1

With conda numpy 1.14.5 and the following numpy config:

``` python -c "import numpy; print(numpy.show_config())" mkl_info: libraries = ['mkl_rt'] library_dirs = ['C:/ProgramData/Anaconda3\\Library\\lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\include', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\lib', 'C:/ProgramData/Anaconda3\\Library\\include'] blas_mkl_info: libraries = ['mkl_rt'] library_dirs = ['C:/ProgramData/Anaconda3\\Library\\lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\include', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\lib', 'C:/ProgramData/Anaconda3\\Library\\include'] blas_opt_info: libraries = ['mkl_rt'] library_dirs = ['C:/ProgramData/Anaconda3\\Library\\lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\include', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\lib', 'C:/ProgramData/Anaconda3\\Library\\include'] lapack_mkl_info: libraries = ['mkl_rt'] library_dirs = ['C:/ProgramData/Anaconda3\\Library\\lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\include', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\lib', 'C:/ProgramData/Anaconda3\\Library\\include'] lapack_opt_info: libraries = ['mkl_rt'] library_dirs = ['C:/ProgramData/Anaconda3\\Library\\lib'] define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)] include_dirs = ['C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\include', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries_2016.4.246\\windows\\mkl\\lib', 'C:/ProgramData/Anaconda3\\Library\\include'] None ```

Is there anything on my end that could still be wrong? Is there a way I can help to resolve this?

[oleksandr-pavlyk]

@TAdeJong Thank for the heads up. I have reproduced the issue on both Windows and Linux. It appears that conda-forge build was built of not the most recent sources due to a fault of mine.

I am in the process of regeneration of the build on conda-forge, see conda-forge/mkl_fft-feedstock#6

I checked that mkl_fft built from the source tar-ball with that sha256 does not exhibit this problem on Linux, but does exhibit it with the build 0 of mkl_fft_1.0.3.

I realize it's a mess, but hopefully it is going to be straightened out soon.

[oleksandr-pavlyk]

@TAdeJong Please retry with 1.0.4 from conda-forge when it becomes available. Should be shortly.

[TAdeJong]

@oleksandr-pavlyk Yes, I can confirm it now works for me on Windows 10 again with version 1.0.4. Thanks for the fast resolution!