Parallel Cythonization for spawned Processes #30862

WillAyd · 2020-01-09T19:33:18Z

Originally thought this only affected Windows, but parallel Cythonization is also broken on Python 3.8 on macOS as they shifted from using fork to start sub-processes to using spawn

https://bugs.python.org/issue33725

This fix should make things work regardless of platform / python version

WillAyd · 2020-01-09T19:34:15Z

@Dr-Irv @topper-123

Dr-Irv · 2020-01-09T19:40:48Z

@WillAyd The cython folks saw that Python 3.8 on macos issue here: conda-forge/cython-feedstock#59 (and maybe elsewhere)

Once you are merged, ping me, and I'll test on Windows.

topper-123 · 2020-01-09T20:37:05Z

I've tried this out on windows and it works for me. I can't however see if the build is running in parallel, maybe @Dr-Irv can comment on that?

Dr-Irv · 2020-01-09T21:26:18Z

I now learned how to fetch this pull request and tried it out. Definitely goes parallel (I use -j 8 since I have 8 real cores). BUT there is an issue that some of the compiles of the C code are not finished before the final "cl" step. So that step fails, and I have to do python setup.py build_ext --inplace -j 8 again and then it works.

WillAyd · 2020-01-09T21:33:35Z

What do you mean by "cl" step? Also is it failing during cythonize or during generation of the libraries?

Dr-Irv · 2020-01-09T21:37:42Z

When I try a clean build, here is the end of what is on the screen:

C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Anaconda3\envs\pandas-dev\libs /LIBPATH:C:\Anaconda3\envs\pandas-dev\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.23.28105\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" /EXPORT:PyInit_json build\temp.win-amd64-3.7\Release\pandas/_libs/src/ujson/python/ujson.obj build\temp.win-amd64-3.7\Release\pandas/_libs/src/ujson/python/objToJSON.obj build\temp.win-amd64-3.7\Release\pandas/_libs/src/ujson/python/JSONtoObj.obj build\temp.win-amd64-3.7\Release\pandas/_libs/src/ujson/lib/ultrajsonenc.obj build\temp.win-amd64-3.7\Release\pandas/_libs/src/ujson/lib/ultrajsondec.obj build\temp.win-amd64-3.7\Release\pandas/_libs/tslibs/src/datetime/np_datetime.obj build\temp.win-amd64-3.7\Release\pandas/_libs/tslibs/src/datetime/np_datetime_strings.obj /OUT:c:\Code\pandas_dev\pandas\pandas\_libs\json.cp37-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.7\Release\pandas/_libs/src/ujson/python\json.cp37-win_amd64.lib
   Creating library build\temp.win-amd64-3.7\Release\pandas/_libs/src/ujson/python\json.cp37-win_amd64.lib and object build\temp.win-amd64-3.7\Release\pandas/_libs/src/ujson/python\json.cp37-win_amd64.exp
Generating code
c:\Code\pandas_dev\pandas\pandas\_libs\src\ujson\python\objToJSON.c(181) : warning C4715: 'initObjToJSON': not all control paths return a value
Finished generating code
Finished generating code
Finished generating code
Finished generating code
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.23.28105\\bin\\HostX86\\x64\\cl.exe' failed with exit status 1

So in the "link" part, the link is failing because it is still generating code for other modules. The "cl" part is the compilation that is being done of each module.

WillAyd · 2020-01-09T21:39:45Z

Do you actually get an error message anywhere besides the ultimate exit status? Are you seeing that with -j4?

WillAyd · 2020-01-09T21:40:52Z

Also note that this only affects the Cythonize piece (i.e. generating C code from the pyx modules) so I'm not sure that would be related to linker errors

Dr-Irv · 2020-01-09T21:51:19Z

With -j 4, I don't get the error. So I think that the -j 8 doesn't work when it is trying to build the final extensions, after the cythonize step.

Dr-Irv · 2020-01-09T22:29:36Z

So I did see this in the output:

c:\Code\pandas_dev\pandas\pandas\_libs\tslibs\src\datetime\np_datetime.c : fatal error C1083: Cannot open compiler generated file: 'c:\Code\pandas_dev\pandas\build\temp.win-amd64-3.7\Release\pandas\_libs\tslibs\src\datetime\np_datetime.obj': Permission denied

I think there is a parallelization conflict in that the same file is being compiled by two different processes. Investigating this

jreback · 2020-01-10T13:41:42Z

can we confirm this doesn't break things with -j 8?

Dr-Irv · 2020-01-10T14:29:17Z

@jreback The cythonization part with -j 8 works in parallel just fine. It's the compilation/linking of the extensions from C to binary that is broken with -j 8 which is why I opened #30873

WillAyd · 2020-01-10T15:43:22Z

@Dr-Irv just to confirm you get that same issue on master with -j8 right?

Dr-Irv · 2020-01-10T15:52:57Z

@Dr-Irv just to confirm you get that same issue on master with -j8 right?

Yes. That's why I opened up #30873

…comp

jreback · 2020-01-20T19:36:55Z

is this a common pattern in setup.py?

WillAyd · 2020-01-20T20:24:06Z

What is your question in reference to? The if __name__ == "__main__" guard or enabling multiprocessing?

jreback · 2020-01-20T20:34:27Z

What is your question in reference to? The if __name__ == "__main__" guard or enabling multiprocessing?

both - iow is this the standard for parallel building

if not why are we deviating ?

WillAyd · 2020-01-20T20:47:19Z

It might help to separate the conversation of building from the conversation of cythonizing. This PR deals only with the latter

Cython uses the multiprocessing module to cythonize individual files. multiprocessing in turn requires the freeze_support call as the very first call in a if __name__ == "__main__" guard for systems which use the spawn method for starting a new process. Traditionally this was just Windows, but as of Python 3.8 it covers macOS as well

jreback · 2020-01-20T22:24:44Z

setup.py

-)
+if __name__ == "__main__":
+    # Freeze to support parallel compilation when using spawn instead of fork
+    multiprocessing.freeze_support()


why does this need to be here, why wouldn't you just put this at the top;

the checking if __main__ seems very odd (in setup.py), I have never seen this, can you point to an authoritative reference for this.

Just for some other examples - numpy does something like this:

https://github.com/numpy/numpy/blob/a16dfb5b397718c5caf32c63f024d1ce3518c5a1/setup.py#L484

likewise with scikit-learn:

https://github.com/scikit-learn/scikit-learn/blob/00fe3d6944f91d52b24d0f59cc9a4dd83be99bcf/setup.py#L302

I can move to the top but I think would be weird to include everything in that; especially with everything that is currently global

gotcha, can you change to the setup_package pattern

Sounds good - just did in latest. We probably could/should make it so that there aren't any global variables in the setup file, but it would be a rather large undertaking so kept most globals where they are for now

…comp

jreback · 2020-01-24T04:01:20Z

thanks @WillAyd

WillAyd added 2 commits January 9, 2020 11:31

Parallel compilation for windows

292f941

removed unneeded check

2ca1b03

gfyoung added Bug Build Library building on various platforms Enhancement and removed Bug labels Jan 9, 2020

Dr-Irv mentioned this pull request Jan 9, 2020

Parallel build_ext might fail due to file collisions #30873

Closed

Merge remote-tracking branch 'upstream/master' into windows-parallel-…

82336eb

…comp

jreback added this to the 1.1 milestone Jan 20, 2020

jreback reviewed Jan 20, 2020

View reviewed changes

WillAyd added 2 commits January 20, 2020 15:34

Merge remote-tracking branch 'upstream/master' into windows-parallel-…

1eeb889

…comp

setup_package func

64db610

WillAyd closed this Jan 21, 2020

WillAyd reopened this Jan 21, 2020

jreback merged commit 754dc4c into pandas-dev:master Jan 24, 2020

WillAyd deleted the windows-parallel-comp branch April 12, 2023 20:17

Uh oh!

Parallel Cythonization for spawned Processes #30862

Parallel Cythonization for spawned Processes #30862

Uh oh!

Conversation

WillAyd commented Jan 9, 2020

Uh oh!

WillAyd commented Jan 9, 2020

Uh oh!

Dr-Irv commented Jan 9, 2020

Uh oh!

topper-123 commented Jan 9, 2020

Uh oh!

Dr-Irv commented Jan 9, 2020

Uh oh!

WillAyd commented Jan 9, 2020

Uh oh!

Dr-Irv commented Jan 9, 2020

Uh oh!

WillAyd commented Jan 9, 2020

Uh oh!

WillAyd commented Jan 9, 2020

Uh oh!

Dr-Irv commented Jan 9, 2020

Uh oh!

Dr-Irv commented Jan 9, 2020

Uh oh!

jreback commented Jan 10, 2020

Uh oh!

Dr-Irv commented Jan 10, 2020

Uh oh!

WillAyd commented Jan 10, 2020

Uh oh!

Dr-Irv commented Jan 10, 2020

Uh oh!

jreback commented Jan 20, 2020

Uh oh!

WillAyd commented Jan 20, 2020

Uh oh!

jreback commented Jan 20, 2020

Uh oh!

WillAyd commented Jan 20, 2020

Uh oh!

jreback Jan 20, 2020

Choose a reason for hiding this comment

Uh oh!

WillAyd Jan 20, 2020

Choose a reason for hiding this comment

Uh oh!

jreback Jan 20, 2020

Choose a reason for hiding this comment

Uh oh!

WillAyd Jan 20, 2020

Choose a reason for hiding this comment

Uh oh!

jreback commented Jan 24, 2020

Uh oh!

Uh oh!