CI/BLD: Flaky ImportError undefined symbol: pandas_datetime_to_datetimestruct

[mroeschke]

This ImportError has been appearing in a few builds lately e.g.

https://github.com/pandas-dev/pandas/runs/6819149053?check_suite_focus=true
https://github.com/pandas-dev/pandas/runs/6819297885?check_suite_focus=true

2022-06-10T05:56:44.5012047Z    Traceback (most recent call last):
2022-06-10T05:56:44.5012729Z      File "/usr/share/miniconda/envs/pandas-dev/lib/python3.8/site-packages/asv/benchmark.py", line 1435, in <module>
2022-06-10T05:56:44.5013162Z        main()
2022-06-10T05:56:44.5013705Z      File "/usr/share/miniconda/envs/pandas-dev/lib/python3.8/site-packages/asv/benchmark.py", line 1428, in main
2022-06-10T05:56:44.5014100Z        commands[mode](args)
2022-06-10T05:56:44.5015119Z      File "/usr/share/miniconda/envs/pandas-dev/lib/python3.8/site-packages/asv/benchmark.py", line 1103, in main_discover
2022-06-10T05:56:44.5015666Z        list_benchmarks(benchmark_dir, fp)
2022-06-10T05:56:44.5016169Z      File "/usr/share/miniconda/envs/pandas-dev/lib/python3.8/site-packages/asv/benchmark.py", line 1088, in list_benchmarks
2022-06-10T05:56:44.5016554Z        for benchmark in disc_benchmarks(root):
2022-06-10T05:56:44.5017076Z      File "/usr/share/miniconda/envs/pandas-dev/lib/python3.8/site-packages/asv/benchmark.py", line 985, in disc_benchmarks
2022-06-10T05:56:44.5018181Z        for module in disc_modules(root_name, ignore_import_errors=ignore_import_errors):
2022-06-10T05:56:44.5018904Z      File "/usr/share/miniconda/envs/pandas-dev/lib/python3.8/site-packages/asv/benchmark.py", line 967, in disc_modules
2022-06-10T05:56:44.5019725Z        for item in disc_modules(name, ignore_import_errors=ignore_import_errors):
2022-06-10T05:56:44.5020409Z      File "/usr/share/miniconda/envs/pandas-dev/lib/python3.8/site-packages/asv/benchmark.py", line 950, in disc_modules
2022-06-10T05:56:44.5020821Z        module = import_module(module_name)
2022-06-10T05:56:44.5021709Z      File "/usr/share/miniconda/envs/pandas-dev/lib/python3.8/importlib/__init__.py", line 127, in import_module
2022-06-10T05:56:44.5022165Z        return _bootstrap._gcd_import(name[level:], package, level)
2022-06-10T05:56:44.5022557Z      File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
2022-06-10T05:56:44.5022982Z      File "<frozen importlib._bootstrap>", line 991, in _find_and_load
2022-06-10T05:56:44.5023378Z      File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
2022-06-10T05:56:44.5023914Z      File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
2022-06-10T05:56:44.5024302Z      File "<frozen importlib._bootstrap_external>", line 843, in exec_module
2022-06-10T05:56:44.5024698Z      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2022-06-10T05:56:44.5025141Z      File "/home/runner/work/pandas/pandas/asv_bench/benchmarks/algorithms.py", line 5, in <module>
2022-06-10T05:56:44.5025505Z        import pandas as pd
2022-06-10T05:56:44.5026003Z      File "/home/runner/work/pandas/pandas/pandas/__init__.py", line 22, in <module>
2022-06-10T05:56:44.5026696Z        from pandas.compat import is_numpy_dev as _is_numpy_dev
2022-06-10T05:56:44.5027294Z      File "/home/runner/work/pandas/pandas/pandas/compat/__init__.py", line 15, in <module>
2022-06-10T05:56:44.5027968Z        from pandas.compat.numpy import (
2022-06-10T05:56:44.5028331Z      File "/home/runner/work/pandas/pandas/pandas/compat/numpy/__init__.py", line 4, in <module>
2022-06-10T05:56:44.5028851Z        from pandas.util.version import Version
2022-06-10T05:56:44.5029223Z      File "/home/runner/work/pandas/pandas/pandas/util/__init__.py", line 1, in <module>
2022-06-10T05:56:44.5030080Z        from pandas.util._decorators import (  # noqa:F401
2022-06-10T05:56:44.5030653Z      File "/home/runner/work/pandas/pandas/pandas/util/_decorators.py", line 14, in <module>
2022-06-10T05:56:44.5050795Z        from pandas._libs.properties import cache_readonly  # noqa:F401
2022-06-10T05:56:44.5051218Z      File "/home/runner/work/pandas/pandas/pandas/_libs/__init__.py", line 13, in <module>
2022-06-10T05:56:44.5051586Z        from pandas._libs.interval import Interval
2022-06-10T05:56:44.5051939Z      File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
2022-06-10T05:56:44.5052312Z      File "pandas/_libs/hashtable.pyx", line 1, in init pandas._libs.hashtable
2022-06-10T05:56:44.5052693Z      File "pandas/_libs/missing.pyx", line 1, in init pandas._libs.missing
2022-06-10T05:56:44.5053108Z      File "/home/runner/work/pandas/pandas/pandas/_libs/tslibs/__init__.py", line 35, in <module>
2022-06-10T05:56:44.5053514Z        from pandas._libs.tslibs.conversion import localize_pydatetime
2022-06-10T05:56:44.5054108Z      File "pandas/_libs/tslibs/conversion.pyx", line 69, in init pandas._libs.tslibs.conversion
2022-06-10T05:56:44.5054557Z      File "pandas/_libs/tslibs/parsing.pyx", line 1, in init pandas._libs.tslibs.parsing
2022-06-10T05:56:44.5055164Z      File "pandas/_libs/tslibs/offsets.pyx", line 1, in init pandas._libs.tslibs.offsets
2022-06-10T05:56:44.5056426Z    ImportError: /home/runner/work/pandas/pandas/pandas/_libs/tslibs/timestamps.cpython-38-x86_64-linux-gnu.so: undefined symbol: pandas_datetime_to_datetimestruct

The flakiness might indicate that the parallel build process is not building everything correctly?

cc @jbrockmendel @lithomas1 @WillAyd

[rhshadrach]

Tried some debugging here: #47310 (comment)

When the build fails, symbols in timestamps.so are unresolved but exist in np_datetime.o (and np_datetime.so). Is it possible the build of timestamps.so is occurring before np_datetime.o exists (or is fully built)?

[MarcoGorelli]

locally I run

while true; do
    python setup.py build_ext -j 16 --force && python -c 'import pandas'
    if [ $? -eq 0 ]; then
        break
    fi
done

, it usually takes a few attempts to build correctly but it's faster than -j 1

[rhshadrach]

Minor tweak to avoid compiling when you don't need to:

python setup.py build_ext -j 16 && python -c 'import pandas';
while [ $? -ne 0 ]; do
    python setup.py build_ext -j 16 --force && python -c 'import pandas';
done