fatal error when building the OpenBLAS ThunderX2 and some other dynamic kernels for Windows on ARM

[hmartinez82]

I gave it another try on building OpenBLAS with DYNAMIC_ARCH=ON (see https://github.com/msys2/MINGW-packages/blob/b07f75117ed0cfe6e1f000de7caaffb433cb8c80/mingw-w64-openblas/PKGBUILD#L72) for Windows on ARM after patching Clang with #67894
Is it possible that I also have to add another patch for supporting the whole fix or this something new related only to ThunderX2?

Clang is still crashing.

[9616/15543] Building C object kernel/CMakeFiles/kernel_THUNDERX2T99.dir/CMakeFiles/dcopy_k_THUNDERX2T99.c.obj
FAILED: kernel/CMakeFiles/kernel_THUNDERX2T99.dir/CMakeFiles/dcopy_k_THUNDERX2T99.c.obj
C:\msys64\clangarm64\bin\clang.exe  -IC:/M_P/mingw-w64-openblas/src/OpenBLAS-0.3.24/lapack-netlib/LAPACKE/include -IC:/M_P/mingw-w64-openblas/src/OpenBLAS-0.3.24 -IC:/M_P/mingw-w64-openblas/src/build-CLANGARM64-32/kernel_config/THUNDERX2T99 -Wno-unused-function -Wno-unused-variable  -DHAVE_C11 -DMS_ABI -fopenmp=libomp -DUSE_OPENMP -Wall -DF_INTERFACE_GFORT -DDYNAMIC_ARCH -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=64 -DMAX_PARALLEL_NUMBER=1 -DNO_AFFINITY -DVERSION="\"0.3.24\"" -DBUILD_SINGLE -DBUILD_DOUBLE -DBUILD_COMPLEX -DBUILD_COMPLEX16 -O3 -DNDEBUG -DBUILD_KERNEL -DTABLE_NAME=gotoblas_THUNDERX2T99  -DTS=_THUNDERX2T99 -MD -MT kernel/CMakeFiles/kernel_THUNDERX2T99.dir/CMakeFiles/dcopy_k_THUNDERX2T99.c.obj -MF kernel\CMakeFiles\kernel_THUNDERX2T99.dir\CMakeFiles\dcopy_k_THUNDERX2T99.c.obj.d -o kernel/CMakeFiles/kernel_THUNDERX2T99.dir/CMakeFiles/dcopy_k_THUNDERX2T99.c.obj -c C:/M_P/mingw-w64-openblas/src/build-CLANGARM64-32/kernel/CMakeFiles/dcopy_k_THUNDERX2T99.c
fatal error: error in backend: Failed to evaluate function length in SEH unwind info
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: C:\\msys64\\clangarm64\\bin\\clang.exe -IC:/M_P/mingw-w64-openblas/src/OpenBLAS-0.3.24/lapack-netlib/LAPACKE/include -IC:/M_P/mingw-w64-openblas/src/OpenBLAS-0.3.24 -IC:/M_P/mingw-w64-openblas/src/build-CLANGARM64-32/kernel_config/THUNDERX2T99 -Wno-unused-function -Wno-unused-variable -DHAVE_C11 -DMS_ABI -fopenmp=libomp -DUSE_OPENMP -Wall -DF_INTERFACE_GFORT -DDYNAMIC_ARCH -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=64 -DMAX_PARALLEL_NUMBER=1 -DNO_AFFINITY -DVERSION=\"0.3.24\" -DBUILD_SINGLE -DBUILD_DOUBLE -DBUILD_COMPLEX -DBUILD_COMPLEX16 -O3 -DNDEBUG -DBUILD_KERNEL -DTABLE_NAME=gotoblas_THUNDERX2T99 -DTS=_THUNDERX2T99 -MD -MT kernel/CMakeFiles/kernel_THUNDERX2T99.dir/CMakeFiles/dcopy_k_THUNDERX2T99.c.obj -MF kernel\\CMakeFiles\\kernel_THUNDERX2T99.dir\\CMakeFiles\\dcopy_k_THUNDERX2T99.c.obj.d -o kernel/CMakeFiles/kernel_THUNDERX2T99.dir/CMakeFiles/dcopy_k_THUNDERX2T99.c.obj -c C:/M_P/mingw-w64-openblas/src/build-CLANGARM64-32/kernel/CMakeFiles/dcopy_k_THUNDERX2T99.c
1.      <eof> parser at end of file
2.      Code generation
3.      Running pass 'Function Pass Manager' on module 'C:/M_P/mingw-w64-openblas/src/build-CLANGARM64-32/kernel/CMakeFiles/dcopy_k_THUNDERX2T99.c'.
4.      Running pass 'AArch64 Assembly Printer' on function '@dcopy_k_THUNDERX2T99'
Exception Code: 0xE0000046
 #0 0x00007fff9ff06334 (C:\Windows\System32\KERNELBASE.dll+0x76334)
 #1 0x00007ffef8f85ad0 llvm::CrashRecoveryContext::HandleExit(int) (C:\msys64\clangarm64\bin\libLLVM-17.dll+0x85ad0)
 #2 0x00007ffef9029ff4 llvm::sys::Process::Exit(int, bool) (C:\msys64\clangarm64\bin\libLLVM-17.dll+0x129ff4)
 #3 0x00007ff7519b6fe8 llvm::InitializeAllAsmPrinters() (C:\msys64\clangarm64\bin\clang.exe+0x6fe8)
 #4 0x00007ffef8f9323c llvm::report_fatal_error(llvm::Twine const&, bool) (C:\msys64\clangarm64\bin\libLLVM-17.dll+0x9323c)
 #5 0x00007ffef8f9311c llvm::report_fatal_error(char const*, bool) (C:\msys64\clangarm64\bin\libLLVM-17.dll+0x9311c)
 #6 0x00007ffefa5e8f4c llvm::Win64EH::ARM64UnwindEmitter::Emit(llvm::MCStreamer&) const (C:\msys64\clangarm64\bin\libLLVM-17.dll+0x16e8f4c)
 #7 0x00007ffefa5de8d4 llvm::MCStreamer::emitWinCFIEndProc(llvm::SMLoc) (C:\msys64\clangarm64\bin\libLLVM-17.dll+0x16de8d4)
 #8 0x00007ffef9938478 llvm::WinException::endFuncletImpl() (C:\msys64\clangarm64\bin\libLLVM-17.dll+0xa38478)
 #9 0x00007ffef99380a8 llvm::WinException::endFunction(llvm::MachineFunction const*) (C:\msys64\clangarm64\bin\libLLVM-17.dll+0xa380a8)
#10 0x00007ffef98d848c llvm::AsmPrinter::emitFunctionBody() (C:\msys64\clangarm64\bin\libLLVM-17.dll+0x9d848c)
clang: error: clang frontend command failed with exit code 70 (use -v to see invocation)
clang version 17.0.2
Target: aarch64-w64-windows-gnu
Thread model: posix
InstalledDir: C:/msys64/clangarm64/bin
clang: note: diagnostic msg:
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang: note: diagnostic msg: C:/msys64/tmp/dcopy_k_THUNDERX2T99-ce9e97.c
clang: note: diagnostic msg: C:/msys64/tmp/dcopy_k_THUNDERX2T99-ce9e97.sh
clang: note: diagnostic msg:

********************
ninja: build stopped: subcommand failed.
==> ERROR: A failure occurred in build().
    Aborting...

[hmartinez82]

dcopy_k_THUNDERX2T99-ce9e97.zip
Here are the two files that Clang says to provide

[llvmbot]

@llvm/issue-subscribers-backend-aarch64

Author: Hernan Martinez (hmartinez82)

I gave it another try on building OpenBLAS with DYNAMIC_ARCH=ON (see https://github.com/msys2/MINGW-packages/blob/b07f75117ed0cfe6e1f000de7caaffb433cb8c80/mingw-w64-openblas/PKGBUILD#L72) for Windows on ARM after patching Clang with https://github.com//pull/67894 Is it possible that I also have to add another patch for supporting the whole fix or this something new related only to ThunderX2?

Clang is still crashing.

[9616/15543] Building C object kernel/CMakeFiles/kernel_THUNDERX2T99.dir/CMakeFiles/dcopy_k_THUNDERX2T99.c.obj
FAILED: kernel/CMakeFiles/kernel_THUNDERX2T99.dir/CMakeFiles/dcopy_k_THUNDERX2T99.c.obj
C:\msys64\clangarm64\bin\clang.exe  -IC:/M_P/mingw-w64-openblas/src/OpenBLAS-0.3.24/lapack-netlib/LAPACKE/include -IC:/M_P/mingw-w64-openblas/src/OpenBLAS-0.3.24 -IC:/M_P/mingw-w64-openblas/src/build-CLANGARM64-32/kernel_config/THUNDERX2T99 -Wno-unused-function -Wno-unused-variable  -DHAVE_C11 -DMS_ABI -fopenmp=libomp -DUSE_OPENMP -Wall -DF_INTERFACE_GFORT -DDYNAMIC_ARCH -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=64 -DMAX_PARALLEL_NUMBER=1 -DNO_AFFINITY -DVERSION="\"0.3.24\"" -DBUILD_SINGLE -DBUILD_DOUBLE -DBUILD_COMPLEX -DBUILD_COMPLEX16 -O3 -DNDEBUG -DBUILD_KERNEL -DTABLE_NAME=gotoblas_THUNDERX2T99  -DTS=_THUNDERX2T99 -MD -MT kernel/CMakeFiles/kernel_THUNDERX2T99.dir/CMakeFiles/dcopy_k_THUNDERX2T99.c.obj -MF kernel\CMakeFiles\kernel_THUNDERX2T99.dir\CMakeFiles\dcopy_k_THUNDERX2T99.c.obj.d -o kernel/CMakeFiles/kernel_THUNDERX2T99.dir/CMakeFiles/dcopy_k_THUNDERX2T99.c.obj -c C:/M_P/mingw-w64-openblas/src/build-CLANGARM64-32/kernel/CMakeFiles/dcopy_k_THUNDERX2T99.c
fatal error: error in backend: Failed to evaluate function length in SEH unwind info
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: C:\\msys64\\clangarm64\\bin\\clang.exe -IC:/M_P/mingw-w64-openblas/src/OpenBLAS-0.3.24/lapack-netlib/LAPACKE/include -IC:/M_P/mingw-w64-openblas/src/OpenBLAS-0.3.24 -IC:/M_P/mingw-w64-openblas/src/build-CLANGARM64-32/kernel_config/THUNDERX2T99 -Wno-unused-function -Wno-unused-variable -DHAVE_C11 -DMS_ABI -fopenmp=libomp -DUSE_OPENMP -Wall -DF_INTERFACE_GFORT -DDYNAMIC_ARCH -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=64 -DMAX_PARALLEL_NUMBER=1 -DNO_AFFINITY -DVERSION=\"0.3.24\" -DBUILD_SINGLE -DBUILD_DOUBLE -DBUILD_COMPLEX -DBUILD_COMPLEX16 -O3 -DNDEBUG -DBUILD_KERNEL -DTABLE_NAME=gotoblas_THUNDERX2T99 -DTS=_THUNDERX2T99 -MD -MT kernel/CMakeFiles/kernel_THUNDERX2T99.dir/CMakeFiles/dcopy_k_THUNDERX2T99.c.obj -MF kernel\\CMakeFiles\\kernel_THUNDERX2T99.dir\\CMakeFiles\\dcopy_k_THUNDERX2T99.c.obj.d -o kernel/CMakeFiles/kernel_THUNDERX2T99.dir/CMakeFiles/dcopy_k_THUNDERX2T99.c.obj -c C:/M_P/mingw-w64-openblas/src/build-CLANGARM64-32/kernel/CMakeFiles/dcopy_k_THUNDERX2T99.c
1.      <eof> parser at end of file
2.      Code generation
3.      Running pass 'Function Pass Manager' on module 'C:/M_P/mingw-w64-openblas/src/build-CLANGARM64-32/kernel/CMakeFiles/dcopy_k_THUNDERX2T99.c'.
4.      Running pass 'AArch64 Assembly Printer' on function '@<!-- -->dcopy_k_THUNDERX2T99'
Exception Code: 0xE0000046
 #<!-- -->0 0x00007fff9ff06334 (C:\Windows\System32\KERNELBASE.dll+0x76334)
 #<!-- -->1 0x00007ffef8f85ad0 llvm::CrashRecoveryContext::HandleExit(int) (C:\msys64\clangarm64\bin\libLLVM-17.dll+0x85ad0)
 #<!-- -->2 0x00007ffef9029ff4 llvm::sys::Process::Exit(int, bool) (C:\msys64\clangarm64\bin\libLLVM-17.dll+0x129ff4)
 #<!-- -->3 0x00007ff7519b6fe8 llvm::InitializeAllAsmPrinters() (C:\msys64\clangarm64\bin\clang.exe+0x6fe8)
 #<!-- -->4 0x00007ffef8f9323c llvm::report_fatal_error(llvm::Twine const&amp;, bool) (C:\msys64\clangarm64\bin\libLLVM-17.dll+0x9323c)
 #<!-- -->5 0x00007ffef8f9311c llvm::report_fatal_error(char const*, bool) (C:\msys64\clangarm64\bin\libLLVM-17.dll+0x9311c)
 #<!-- -->6 0x00007ffefa5e8f4c llvm::Win64EH::ARM64UnwindEmitter::Emit(llvm::MCStreamer&amp;) const (C:\msys64\clangarm64\bin\libLLVM-17.dll+0x16e8f4c)
 #<!-- -->7 0x00007ffefa5de8d4 llvm::MCStreamer::emitWinCFIEndProc(llvm::SMLoc) (C:\msys64\clangarm64\bin\libLLVM-17.dll+0x16de8d4)
 #<!-- -->8 0x00007ffef9938478 llvm::WinException::endFuncletImpl() (C:\msys64\clangarm64\bin\libLLVM-17.dll+0xa38478)
 #<!-- -->9 0x00007ffef99380a8 llvm::WinException::endFunction(llvm::MachineFunction const*) (C:\msys64\clangarm64\bin\libLLVM-17.dll+0xa380a8)
#<!-- -->10 0x00007ffef98d848c llvm::AsmPrinter::emitFunctionBody() (C:\msys64\clangarm64\bin\libLLVM-17.dll+0x9d848c)
clang: error: clang frontend command failed with exit code 70 (use -v to see invocation)
clang version 17.0.2
Target: aarch64-w64-windows-gnu
Thread model: posix
InstalledDir: C:/msys64/clangarm64/bin
clang: note: diagnostic msg:
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang: note: diagnostic msg: C:/msys64/tmp/dcopy_k_THUNDERX2T99-ce9e97.c
clang: note: diagnostic msg: C:/msys64/tmp/dcopy_k_THUNDERX2T99-ce9e97.sh
clang: note: diagnostic msg:

********************
ninja: build stopped: subcommand failed.
==&gt; ERROR: A failure occurred in build().
    Aborting...

[martin-frbg]

Is there something I could do on the OpenBLAS side to help move this along, or some workaround that would allow me to supply a suitable estimate for whatever clang needs to compute the function length here ? The error persists with 19.1

[hmartinez82]

Turns out that it's actually these cores that are failing to build:

• THUNDERX2T99
• THUNDERX3T110
• NEOVERSEN1
• NEOVERSEV1
• NEOVERSEN2
• ARMV8SVE
• A64FX

and working with these cores:

• CORTEXA53
• CORTEXA57
• CORTEXA72
• CORTEXA73
• CORTEXA76
• CORTEXX1
• THUNDERX
• TSV110
• EMAG8180

Here's a table of working and crashing compiler flags:

working	crashing
-march=armv8-a -mtune=cortex-a53	-march=armv8.1-a -mtune=thunderx2t99
-march=armv8-a -mtune=cortex-a57	-march=armv8.3-a -mtune=thunderx2t99
-march=armv8-a -mtune=cortex-a72	-march=armv8.2-a -mtune=neoverse-n1
-march=armv8-a -mtune=cortex-a73	-march=armv8.4-a+sve -mtune=cortex-x1
-march=armv8.2-a -mtune=cortex-a76	-march=armv8.2-a+sve+bf16 -mtune=cortex-a72
-march=armv8.2-a -mtune=cortex-x1	-march=armv8-a+sve
-march=armv8-a -mtune=thunderx	-march=armv8.2-a+sve -mtune=a64fx
-march=armv8.2-a -mtune=tsv110	-march=armv8.4-a+sve
-march=armv8-a

[hmartinez82]

@DavidSpickett @efriedma-quic Do you think you guys can try another look at this?
I'm now trying this with 19.1.6 1.5 years later

[2/579] Building C object kernel/CMakeFiles/kernel_THUNDERX2T99.dir/CMakeFiles/sasum_k_THUNDERX2T99.c.obj
FAILED: kernel/CMakeFiles/kernel_THUNDERX2T99.dir/CMakeFiles/sasum_k_THUNDERX2T99.c.obj
C:\msys64\clangarm64\bin\clang.exe  -IC:/Dev/Github/MINGW-packages/mingw-w64-openblas/src/OpenBLAS-0.3.28/lapack-netlib/LAPACKE/include -IC:/Dev/Github/MINGW-packages/mingw-w64-openblas/src/OpenBLAS-0.3.28 -IC:/Dev/Github/MINGW-packages/mingw-w64-openblas/src/build-CLANGARM64-32/kernel_config/THUNDERX2T99 -DHAVE_C11 -DMS_ABI -fopenmp=libomp -DUSE_OPENMP -Wall -DF_INTERFACE_GFORT -DGEMM_GEMV_FORWARD -DDYNAMIC_ARCH -DDYNAMIC_LIST -DDYN_CORTEXA53 -DDYN_THUNDERX2T99 -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=64 -DMAX_PARALLEL_NUMBER=1 -DNO_AFFINITY -DVERSION="\"0.3.28\"" -DBUILD_SINGLE -DBUILD_DOUBLE -DBUILD_COMPLEX -DBUILD_COMPLEX16 -O3 -DNDEBUG -DBUILD_KERNEL -DTABLE_NAME=gotoblas_THUNDERX2T99  -DTS=_THUNDERX2T99 -MD -MT kernel/CMakeFiles/kernel_THUNDERX2T99.dir/CMakeFiles/sasum_k_THUNDERX2T99.c.obj -MF kernel\CMakeFiles\kernel_THUNDERX2T99.dir\CMakeFiles\sasum_k_THUNDERX2T99.c.obj.d -o kernel/CMakeFiles/kernel_THUNDERX2T99.dir/CMakeFiles/sasum_k_THUNDERX2T99.c.obj -c C:/Dev/Github/MINGW-packages/mingw-w64-openblas/src/build-CLANGARM64-32/kernel/CMakeFiles/sasum_k_THUNDERX2T99.c
fatal error: error in backend: Failed to evaluate function length in SEH unwind info
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: C:\\msys64\\clangarm64\\bin\\clang.exe -IC:/Dev/Github/MINGW-packages/mingw-w64-openblas/src/OpenBLAS-0.3.28/lapack-netlib/LAPACKE/include -IC:/Dev/Github/MINGW-packages/mingw-w64-openblas/src/OpenBLAS-0.3.28 -IC:/Dev/Github/MINGW-packages/mingw-w64-openblas/src/build-CLANGARM64-32/kernel_config/THUNDERX2T99 -DHAVE_C11 -DMS_ABI -fopenmp=libomp -DUSE_OPENMP -Wall -DF_INTERFACE_GFORT -DGEMM_GEMV_FORWARD -DDYNAMIC_ARCH -DDYNAMIC_LIST -DDYN_CORTEXA53 -DDYN_THUNDERX2T99 -DSMP_SERVER -DNO_WARMUP -DMAX_CPU_NUMBER=64 -DMAX_PARALLEL_NUMBER=1 -DNO_AFFINITY -DVERSION=\"0.3.28\" -DBUILD_SINGLE -DBUILD_DOUBLE -DBUILD_COMPLEX -DBUILD_COMPLEX16 -O3 -DNDEBUG -DBUILD_KERNEL -DTABLE_NAME=gotoblas_THUNDERX2T99 -DTS=_THUNDERX2T99 -MD -MT kernel/CMakeFiles/kernel_THUNDERX2T99.dir/CMakeFiles/sasum_k_THUNDERX2T99.c.obj -MF kernel\\CMakeFiles\\kernel_THUNDERX2T99.dir\\CMakeFiles\\sasum_k_THUNDERX2T99.c.obj.d -o kernel/CMakeFiles/kernel_THUNDERX2T99.dir/CMakeFiles/sasum_k_THUNDERX2T99.c.obj -c C:/Dev/Github/MINGW-packages/mingw-w64-openblas/src/build-CLANGARM64-32/kernel/CMakeFiles/sasum_k_THUNDERX2T99.c
1.      <eof> parser at end of file
2.      Code generation
3.      Running pass 'Function Pass Manager' on module 'C:/Dev/Github/MINGW-packages/mingw-w64-openblas/src/build-CLANGARM64-32/kernel/CMakeFiles/sasum_k_THUNDERX2T99.c'.
4.      Running pass 'AArch64 Assembly Printer' on function '@sasum_k_THUNDERX2T99'
Exception Code: 0xE0000046
#0 0x00007ffe5a646248 (C:\Windows\System32\KERNELBASE.dll+0xb6248)
#1 0xc73d7ffd70a8d7d8
clang: error: clang frontend command failed with exit code 70 (use -v to see invocation)
clang version 19.1.6
Target: aarch64-w64-windows-gnu
Thread model: posix
InstalledDir: C:/msys64/clangarm64/bin
clang: note: diagnostic msg:
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang: note: diagnostic msg: C:/msys64/tmp/sasum_k_THUNDERX2T99-ee3aec.c
clang: note: diagnostic msg: C:/msys64/tmp/sasum_k_THUNDERX2T99-ee3aec.sh
clang: note: diagnostic msg:

********************
ninja: build stopped: subcommand failed.

sasum_k_THUNDERX2T99-ee3aec.zip

I also got the whole file preprocessed with -E. This is the output, and indeed, there's a function called sasum_k_THUNDERX2T99 towards the end of the file.

preprocessed.zip

[martin-frbg]

@hmartinez82 I'm not sure if your tables are useful in this context - it is most likely not the presence or absence of mtune flags but simply the reuse of originally Thunderx2t99-specific BLAS kernels (with instruction sequences that clang fails to handle) for later cpus

[martin-frbg]

the commonality here seems to be the use of .align directives in the affected assembly files. I guess we could try removing (#ifdef'ing) them to see if the code compiles - though I'm not yet sure what that would do to performance.