LLVM JIT on Windows ARM32 (thumbv7-pc-win32)

[driver1998]

I am trying to bootstrap Mesa LLVMpipe on Windows on ARM, so far ARM64 is working just fine.

But on ARM32 it doesn't seem to work.

A reduced repro (tested on ARM64 machine. LLVM 15.0.7 built as ARM32, with LLVM-MinGW toolchain):

$ cat test.ll
; test.ll
define i32 @main() {
  ret i32 42
}

$ /d/install/bin/lli --version
LLVM (http://llvm.org/):
  LLVM version 15.0.7
  DEBUG build with assertions.
  Default target: armv7-w64-windows-gnu
  Host CPU: (unknown)

$ /d/install/bin/lli test.ll
error: Function 'main' uses ARM instructions, but the target does not support ARM mode execution.

$ /d/install/bin/lli --march=thumbv7 test.ll
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: D:\\install\\bin\\lli.exe --march=thumbv7 test.ll
Exception Code: 0xC0000005
#0 0x1fce1000
#1 0x003ffb72 llvm::orc::runAsMain(int (*)(int, char**), llvm::ArrayRef<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, llvm::Optional<llvm::StringRef>) D:/llvm-project-15.0.7.src/llvm/lib/ExecutionEngine/Orc/TargetProcess/TargetExecutionUtils.cpp:39:10
#2 0x000f5066 runOrcJIT(char const*) D:/llvm-project-15.0.7.src/llvm/tools/lli/lli.cpp:1153:14
#3 0x000f2d88 main D:/llvm-project-15.0.7.src/llvm/tools/lli/lli.cpp:507:12
#4 0x000f13e6 __tmainCRTStartup /build/mingw-w64/mingw-w64-crt/build-armv7/../crt/crtexe.c:329:15
#5 0x000f142a .l_start /build/mingw-w64/mingw-w64-crt/build-armv7/../crt/crtexe.c:206:9
#6 0x77005650 (C:\windows\System32\KERNEL32.DLL+0x5650)
#7 0x770fbad4 (C:\windows\SYSTEM32\ntdll.dll+0x4bad4)
Segmentation fault

$ /d/install/bin/lli --march=thumb test.ll
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: D:\\install\\bin\\lli.exe --march=thumb test.ll
Exception Code: 0xC0000005
#0 0x24581000
#1 0x003ffb72 llvm::orc::runAsMain(int (*)(int, char**), llvm::ArrayRef<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, llvm::Optional<llvm::StringRef>) D:/llvm-project-15.0.7.src/llvm/lib/ExecutionEngine/Orc/TargetProcess/TargetExecutionUtils.cpp:39:10
#2 0x000f5066 runOrcJIT(char const*) D:/llvm-project-15.0.7.src/llvm/tools/lli/lli.cpp:1153:14
#3 0x000f2d88 main D:/llvm-project-15.0.7.src/llvm/tools/lli/lli.cpp:507:12
#4 0x000f13e6 __tmainCRTStartup /build/mingw-w64/mingw-w64-crt/build-armv7/../crt/crtexe.c:329:15
#5 0x000f142a .l_start /build/mingw-w64/mingw-w64-crt/build-armv7/../crt/crtexe.c:206:9
#6 0x77005650 (C:\windows\System32\KERNEL32.DLL+0x5650)
#7 0x770fbad4 (C:\windows\SYSTEM32\ntdll.dll+0x4bad4)
Segmentation fault

$ /d/install/bin/lli --march=thumb --jit-kind=mcjit test.ll
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: D:\\install\\bin\\lli.exe --march=thumb --jit-kind=mcjit test.ll
Exception Code: 0xC0000005
#0 0x1fce1000
#1 0x0030882e llvm::MCJIT::runFunction(llvm::Function*, llvm::ArrayRef<llvm::GenericValue>) D:/llvm-project-15.0.7.src/llvm/lib/ExecutionEngine/MCJIT/MCJIT.cpp:585:37
#2 0x002f5ba0 llvm::ExecutionEngine::runFunctionAsMain(llvm::Function*, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, char const* const*) D:/llvm-project-15.0.7.src/llvm/lib/ExecutionEngine/ExecutionEngine.cpp:467:10
#3 0x000f3b88 main D:/llvm-project-15.0.7.src/llvm/tools/lli/lli.cpp:744:18
#4 0x000f13e6 __tmainCRTStartup /build/mingw-w64/mingw-w64-crt/build-armv7/../crt/crtexe.c:329:15
#5 0x000f142a .l_start /build/mingw-w64/mingw-w64-crt/build-armv7/../crt/crtexe.c:206:9
#6 0x77005650 (C:\windows\System32\KERNEL32.DLL+0x5650)
#7 0x770fbad4 (C:\windows\SYSTEM32\ntdll.dll+0x4bad4)
Segmentation fault

[llvmbot]

@llvm/issue-subscribers-orcjit

[llvmbot]

@llvm/issue-subscribers-backend-arm

[lhames]

@driver1998 What's the output of llc test.ll?

I'm not sure if this is producing a COFF or an ELF object, but either way the support in MCJIT may be incomplete.

The right long term solution is to implement arm32 architecture support and corresponding backends in JITLink, then migrate Mesa llvmpipe to ORC. That's a decent sized project though.

In the short term if we can figure out what object format is being used we might be able to fix RuntimeDyld and keep it running a little longer.

[driver1998]

From Mesa comments:

    /*
     * MCJIT works on Windows, but currently only through ELF object format.
     *
     * XXX: We could use `LLVM_HOST_TRIPLE "-elf"` but LLVM_HOST_TRIPLE has
     * different strings for MinGW/MSVC, so better play it safe and be
     * explicit.
     */

Not sure if it is still relevant, but all currently working targets (i686, x86_64, aarch64) use the ELF format (e.g.: x86_64-pc-win32-elf) .

I'll test llc later today.

[lhames]

Did you have any luck with llc? I suspect that this is a JIT error, but llc behavior may give useful insights.

@weliveindetail just made the arm32 support path much more achievable: https://reviews.llvm.org/D144083 (though porting llvmpipe to ORC may still take some work).

[driver1998]

llc does seem to generate working asm, that generates code for COFF by default, and I successfully build the asm to a working executable with clang.

[lhames]

@driver1998 Thanks very much for that (and apologies for the delayed reply). I think that the ideal solution will be to wait until @weliveindetail's arm32 architecture support lands, then write a COFF_aarch32 JITLink backend.

[weliveindetail]

FYI the initial ELF_aarch32 has landed in https://reviews.llvm.org/rG5c1d160cd979dd6099f4266db600020a0cb4867a