Parallel build_ext might fail due to file collisions

[Dr-Irv]

Code Sample, a copy-pastable example if possible

python setup.py build_ext --inplace -j 8

Problem description

See some discussion in #30862 . I have 8 cores on Windows. When I use -j 8, in the middle of the build, I will get this error message:

c:\Code\pandas_dev\pandas\pandas\_libs\tslibs\src\datetime\np_datetime.c : fatal error C1083: Cannot open compiler generated file: 'c:\Code\pandas_dev\pandas\build\temp.win-amd64-3.7\Release\pandas\_libs\tslibs\src\datetime\np_datetime.obj': Permission denied

What I think is happening is that in setup.py, _libs.tslibs.conversion, libs.tslibs.np_datetime and _libs.tslibs.period all have to compile pandas/_libs/tslibs/src/datetime/np_datetime.c . If the parallel build is timed in such a way that two of those extensions are being built at the same time, then they can conflict in writing np_datetime.obj. A similar thing can happen with _libs.lib, _libs.parsers, and _libs.tslibs.parsing all having to compile pandas/_libs/src/parser/tokenizer.c .

If I do two python setup.py build_ext --inplace -j 8 steps in a row, then everything is fine. Also things are fine with -j 4. But I can imagine that even with -j 4, you could end up with the same kind of issue, even in CI.

I don't know enough about how extensions are built, distutils and setuptools in order to avoid this potential collision during the build process.

Output of pd.show_versions()

c:\Code\pandas_dev\pandas\pandas\core\index.py:29: FutureWarning: pandas.core.index is deprecated and will be removed in a future version. The public classes are available in the top-level namespace.
FutureWarning,

INSTALLED VERSIONS

commit : 13858f6
python : 3.7.6.final.0
python-bits : 64
OS : Windows
OS-release : 10
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 13, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None

pandas : 0.26.0.dev0+1734.g13858f6e1
numpy : 1.17.3
pytz : 2019.3
dateutil : 2.8.1
pip : 19.3.1
setuptools : 42.0.2.post20191201
Cython : 0.29.14
pytest : 5.3.2
hypothesis : 4.57.1
sphinx : 2.3.1
blosc : None
feather : None
xlsxwriter : 1.2.7
lxml.etree : 4.4.2
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.10.3
IPython : 7.10.1
pandas_datareader: None
bs4 : 4.8.2
bottleneck : 1.3.1
fastparquet : 0.3.2
gcsfs : None
lxml.etree : 4.4.2
matplotlib : 3.1.2
numexpr : 2.7.0
odfpy : None
openpyxl : 3.0.1
pandas_gbq : None
pyarrow : 0.15.1
pytables : None
pytest : 5.3.2
s3fs : 0.4.0
scipy : 1.3.1
sqlalchemy : 1.3.12
tables : 3.6.1
tabulate : 0.8.6
xarray : 0.14.1
xlrd : 1.2.0
xlwt : 1.3.0
xlsxwriter : 1.2.7
numba : 0.46.0

[WillAyd]

You are using the MSVC compiler right? Not sure if it matters just asking

[Dr-Irv]

You are using the MSVC compiler right? Not sure if it matters just asking

Yes.

[alimcmaster1]

I sometimes do pandas dev on windows and recall running into this exact problem with -j 4

[alimcmaster1]

Maybe @scoder or someone from Cython can help?

[adamjstewart]

Was this fixed by #30862? Can this be closed now?

[Dr-Irv]

@adamjstewart Don't think so. I opened this one while #30862 was being developed.

[mgorny]

This is actually a pretty awful bug in distutils. Basically, if you use the same source file in two extensions, distutils can hit pretty nasty race conditions compiling the file twice simultaneously. This is especially bad on Linux because there's no immediate error, but the installed extensions are simply broken (e.g. missing symbols). I'm going to submit a PR shortly.

[leycec]

I'm equally astonished, appalled, and disappointed that Pandas developers like @jreback pushed back against #40285 with nonsensical commentary like "why is j more than like 4 actually useful?" I mean, come on. Scalability is an unconditional good. This is not up for meaningful debate. 🤦

The trivial resolution here is to do what everyone else in Python's standard scientific stack does: disable parallel building entirely by forcing -j 1 in setup.py. Yes, that's demonstrably awful – but there are no working alternatives on the table. Forcing -j 1 at C extension build time is exactly what SciPy and everyone else does. That's the MostlyRightThing™ to do, because setuptools upstream has no interest or intention of resolving this within the expected lifetime of our Universe.

Fortunately, this is trivial. Just override the finalize_options() method of your custom build_ext class in setup.py to resemble:

class build_ext(_build_ext):
    def finalize_options(self):
        super().finalize_options()

        # Disable distutils parallel build due to upstream race conditions. See
        # pandas-dev/pandas#30873
        if self.parallel:
            print("NOTE: -j build option not supported.")
        self.parallel = None

Boom! Issue trivially solved. Pandas then happily munch on bamboo shoots as a show of appreciation. 🎍 🐼