"Unimplemented reloc type" when running MLIR mlir-runner based tests on RISC-V Architecture

[vijayaramaraju-kalidindi]

Maintainer note: Originally this was about this build issue but once past that, the issue reflected in the title was discovered. See comments further down for details.

We are following MLIR source code compilation for RISCV architecture shared as per official document.

####################################################################################

root@ubuntu:~/llvm-project-main/build# cmake -G "Ninja"
-DCMAKE_C_COMPILER=$(which clang)
-DCMAKE_CXX_COMPILER=$(which clang++)
-DLLVM_ENABLE_PROJECTS="mlir"
-DLLVM_TARGETS_TO_BUILD="RISCV"
-DCMAKE_BUILD_TYPE="RelWithDebInfo"
-DLLVM_ENABLE_ASSERTIONS=ON
-DLLVM_BUILD_EXAMPLES=ON
-DCMAKE_INSTALL_PREFIX=$(pwd)/../installed
-DLLVM_PARALLEL_COMPILE_JOBS=1
-DLLVM_PARALLEL_LINK_JOBS=1
-DLLVM_BUILD_LLVM_DYLIB=ON
-DLLVM_LINK_LLVM_DYLIB=ON
-DBUILD_SHARED_LIBS=OFF
-DLLVM_USE_SPLIT_DWARF=ON
-DLLVM_OPTIMIZED_TABLEGEN=ON
-DLLVM_USE_NEWPM=ON
-DLLVM_CCACHE_BUILD=OFF
../llvm

Source code compilation is successful in this step.

###################################################################################

Now while building the object code we are facing below error.

root@ubuntu:~/llvm-project-main/build# cmake --build . --target check-mlir
[1/4498] Building CXX object utils/FileCheck/CMakeFiles/FileCheck.dir/FileCheck.cpp.o
FAILED: utils/FileCheck/CMakeFiles/FileCheck.dir/FileCheck.cpp.o
/usr/bin/clang++ -DGTEST_HAS_RTTI=0 -DLLVM_BUILD_STATIC -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/root/llvm-project-main/build/utils/FileCheck -I/root/llvm-project-main/llvm/utils/FileCheck -I/root/llvm-project-main/build/include -I/root/llvm-project-main/llvm/include -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -O2 -g -DNDEBUG -std=c++17 -fno-exceptions -funwind-tables -fno-rtti -UNDEBUG -gsplit-dwarf -MD -MT utils/FileCheck/CMakeFiles/FileCheck.dir/FileCheck.cpp.o -MF utils/FileCheck/CMakeFiles/FileCheck.dir/FileCheck.cpp.o.d -o utils/FileCheck/CMakeFiles/FileCheck.dir/FileCheck.cpp.o -c /root/llvm-project-main/llvm/utils/FileCheck/FileCheck.cpp
clang++: error: -gsplit-dwarf is unsupported with RISC-V linker relaxation (-mrelax)
ninja: build stopped: subcommand failed.

[llvmbot]

@llvm/issue-subscribers-mlir

Author: None (vijayaramaraju-kalidindi)

We are following MLIR source code compilation for RISCV architecture shared as per official document.

####################################################################################

root@ubuntu:~/llvm-project-main/build# cmake -G "Ninja"
-DCMAKE_C_COMPILER=$(which clang)
-DCMAKE_CXX_COMPILER=$(which clang++)
-DLLVM_ENABLE_PROJECTS="mlir"
-DLLVM_TARGETS_TO_BUILD="RISCV"
-DCMAKE_BUILD_TYPE="RelWithDebInfo"
-DLLVM_ENABLE_ASSERTIONS=ON
-DLLVM_BUILD_EXAMPLES=ON
-DCMAKE_INSTALL_PREFIX=$(pwd)/../installed
-DLLVM_PARALLEL_COMPILE_JOBS=1
-DLLVM_PARALLEL_LINK_JOBS=1
-DLLVM_BUILD_LLVM_DYLIB=ON
-DLLVM_LINK_LLVM_DYLIB=ON
-DBUILD_SHARED_LIBS=OFF
-DLLVM_USE_SPLIT_DWARF=ON
-DLLVM_OPTIMIZED_TABLEGEN=ON
-DLLVM_USE_NEWPM=ON
-DLLVM_CCACHE_BUILD=OFF
../llvm

Source code compilation is successful in this step.

###################################################################################

Now while building the object code we are facing below error.

root@ubuntu:~/llvm-project-main/build# cmake --build . --target check-mlir
[1/4498] Building CXX object utils/FileCheck/CMakeFiles/FileCheck.dir/FileCheck.cpp.o
FAILED: utils/FileCheck/CMakeFiles/FileCheck.dir/FileCheck.cpp.o
/usr/bin/clang++ -DGTEST_HAS_RTTI=0 -DLLVM_BUILD_STATIC -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/root/llvm-project-main/build/utils/FileCheck -I/root/llvm-project-main/llvm/utils/FileCheck -I/root/llvm-project-main/build/include -I/root/llvm-project-main/llvm/include -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -O2 -g -DNDEBUG -std=c++17 -fno-exceptions -funwind-tables -fno-rtti -UNDEBUG -gsplit-dwarf -MD -MT utils/FileCheck/CMakeFiles/FileCheck.dir/FileCheck.cpp.o -MF utils/FileCheck/CMakeFiles/FileCheck.dir/FileCheck.cpp.o.d -o utils/FileCheck/CMakeFiles/FileCheck.dir/FileCheck.cpp.o -c /root/llvm-project-main/llvm/utils/FileCheck/FileCheck.cpp
clang++: error: -gsplit-dwarf is unsupported with RISC-V linker relaxation (-mrelax)
ninja: build stopped: subcommand failed.

[DavidSpickett]

That configuration includes some options that are build optimisations rather than required options to make MLIR itself work at all.

In this case, -DLLVM_USE_SPLIT_DWARF=ON is leading to this error. You can read about the effect of that here - https://llvm.org/docs/GettingStarted.html#common-problems.

Turning it off would be fine if you have enough RAM to handle the build.

If not, I think you could add -mno-relax to the flags (https://clang.llvm.org/docs/ClangCommandLineReference.html#cmdoption-clang-mrelax) instead. Though I am no RISC-V expert so I don't know the effect of that on the binaries. I assume they will be functionally the same but a bit slower?

You would add this by adding these options to the cmake command:

-DCMAKE_C_FLAGS="-mno-relax"
-DCMAKE_CXX_FLAGS="-mno-relax"

(https://cmake.org/cmake/help/latest/variable/CMAKE_LANG_FLAGS.html)

I recommend clearing the build directory before running CMake again, just to be sure the new settings are being used.

[DavidSpickett]

That said, we should consider whether the documentation should change to note this problem. Or give a command that would not lead to this problem.

@MaskRay since you wrote https://maskray.me/blog/2021-03-14-the-dark-side-of-riscv-linker-relaxation, do you know why these options are incompatible? Is it something where this only effects one linker or is it a generic problem?

[DavidSpickett]

as per official document.

@vijayaramaraju-kalidindi could you link to the specific document?

A basic Google search gives me https://mlir.llvm.org/getting_started/, which has a simpler cmake command, but I do remember seeing the one you have, in a different document.

[vijayaramaraju-kalidindi]

Hi @DavidSpickett,

I have added some additional flags during compilation.

Let me also try the solution that you have suggested.

[vijayaramaraju-kalidindi]

Hi @DavidSpickett ,

Tried compilation excluding -DLLVM_USE_SPLIT_DWARF=ON flag. Compilation was successful.

But while building the object code facing below issue.

[283/4325] Building IntrinsicsHexagon.h...
FAILED: include/llvm/IR/IntrinsicsHexagon.h /root/llvm-project-main/build/include/llvm/IR/IntrinsicsHexagon.h
cd /root/llvm-project-main/build && /root/llvm-project-main/build/bin/llvm-min-tblgen -gen-intrinsic-enums -intrinsic-prefix=hexagon -I /root/llvm-project-main/llvm/include/llvm/IR -I/root/llvm-project-main/build/include -I/root/llvm-project-main/llvm/include /root/llvm-project-main/llvm/include/llvm/IR/Intrinsics.td --write-if-changed -o include/llvm/IR/IntrinsicsHexagon.h -d include/llvm/IR/IntrinsicsHexagon.h.d
AddressSanitizer:DEADLYSIGNAL
=================================================================
==97177==ERROR: AddressSanitizer: SEGV on unknown address 0x100d5021fc28 (pc 0x555d35a6c6da bp 0x7fffd667e140 sp 0x7fffd667d8e0 T-1)
==97177==The signal is caused by a READ memory access.
AddressSanitizer: CHECK failed: asan_suppressions.cpp:47 "((suppression_ctx)) != (0)" (0x0, 0x0) (tid=97177)
AddressSanitizer: CHECK failed: asan_suppressions.cpp:47 "((suppression_ctx)) != (0)" (0x0, 0x0) (tid=97177)
Illegal instruction
[284/4325] Building IntrinsicsVE.h...
FAILED: include/llvm/IR/IntrinsicsVE.h /root/llvm-project-main/build/include/llvm/IR/IntrinsicsVE.h
cd /root/llvm-project-main/build && /root/llvm-project-main/build/bin/llvm-min-tblgen -gen-intrinsic-enums -intrinsic-prefix=ve -I /root/llvm-project-main/llvm/include/llvm/IR -I/root/llvm-project-main/build/include -I/root/llvm-project-main/llvm/include /root/llvm-project-main/llvm/include/llvm/IR/Intrinsics.td --write-if-changed -o include/llvm/IR/IntrinsicsVE.h -d include/llvm/IR/IntrinsicsVE.h.d
AddressSanitizer:DEADLYSIGNAL
=================================================================
==97198==ERROR: AddressSanitizer: SEGV on unknown address 0x100d5235b828 (pc 0x555a8ffff6da bp 0x7fffe705c150 sp 0x7fffe705b8f0 T-1)
==97198==The signal is caused by a READ memory access.
AddressSanitizer: CHECK failed: asan_suppressions.cpp:47 "((suppression_ctx)) != (0)" (0x0, 0x0) (tid=97198)
AddressSanitizer: CHECK failed: asan_suppressions.cpp:47 "((suppression_ctx)) != (0)" (0x0, 0x0) (tid=97198)
Illegal instruction
[285/4325] Building IntrinsicEnums.inc...
FAILED: include/llvm/IR/IntrinsicEnums.inc /root/llvm-project-main/build/include/llvm/IR/IntrinsicEnums.inc
cd /root/llvm-project-main/build && /root/llvm-project-main/build/bin/llvm-min-tblgen -gen-intrinsic-enums -I /root/llvm-project-main/llvm/include/llvm/IR -I/root/llvm-project-main/build/include -I/root/llvm-project-main/llvm/include /root/llvm-project-main/llvm/include/llvm/IR/Intrinsics.td --write-if-changed -o include/llvm/IR/IntrinsicEnums.inc -d include/llvm/IR/IntrinsicEnums.inc.d
AddressSanitizer:DEADLYSIGNAL
=================================================================
==97164==ERROR: AddressSanitizer: SEGV on unknown address 0x100d53342e28 (pc 0x5557057986da bp 0x7fffeef97160 sp 0x7fffeef96900 T-1)
==97164==The signal is caused by a READ memory access.
AddressSanitizer: CHECK failed: asan_suppressions.cpp:47 "((suppression_ctx)) != (0)" (0x0, 0x0) (tid=97164)
AddressSanitizer: CHECK failed: asan_suppressions.cpp:47 "((suppression_ctx)) != (0)" (0x0, 0x0) (tid=97164)
Illegal instruction
[286/4325] Building IntrinsicImpl.inc...
FAILED: include/llvm/IR/IntrinsicImpl.inc /root/llvm-project-main/build/include/llvm/IR/IntrinsicImpl.inc
cd /root/llvm-project-main/build && /root/llvm-project-main/build/bin/llvm-min-tblgen -gen-intrinsic-impl -I /root/llvm-project-main/llvm/include/llvm/IR -I/root/llvm-project-main/build/include -I/root/llvm-project-main/llvm/include /root/llvm-project-main/llvm/include/llvm/IR/Intrinsics.td --write-if-changed -o include/llvm/IR/IntrinsicImpl.inc -d include/llvm/IR/IntrinsicImpl.inc.d
AddressSanitizer:DEADLYSIGNAL
=================================================================
==97163==ERROR: AddressSanitizer: SEGV on unknown address 0x100d4f4ff428 (pc 0x55602f6f86da bp 0x7fffcfd7a160 sp 0x7fffcfd79900 T-1)
==97163==The signal is caused by a READ memory access.
AddressSanitizer: CHECK failed: asan_suppressions.cpp:47 "((suppression_ctx)) != (0)" (0x0, 0x0) (tid=97163)
AddressSanitizer: CHECK failed: asan_suppressions.cpp:47 "((suppression_ctx)) != (0)" (0x0, 0x0) (tid=97163)
Illegal instruction

Build Logs.txt
Compilation logs removing SPLIT_DWARF.txt

Have attached compilation logs and build logs

[DavidSpickett]

Tried compilation excluding -DLLVM_USE_SPLIT_DWARF=ON flag. Compilation was successful.

Cool.

[285/4325] Building IntrinsicEnums.inc...

These .inc files are built by using the tblgen built at the very start of the build:

cd /root/llvm-project-main/build && /root/llvm-project-main/build/bin/llvm-min-tblgen <...>

So this is the binary failing a sanitizer check.

The failure itself, I am not sure if it's legitimate, or if ASAN itself has done something wrong:

==97164==ERROR: AddressSanitizer: SEGV on unknown address 0x100d53342e28 (pc 0x5557057986da bp 0x7fffeef97160 sp 0x7fffeef96900 T-1)
==97164==The signal is caused by a READ memory access.

Sounds a bit like AddressSanitizer itself caused a SEGV.

AddressSanitizer: CHECK failed: asan_suppressions.cpp:47 "((suppression_ctx)) != (0)" (0x0, 0x0) (tid=97164)
Illegal instruction

Illegal instruction might just be the way AddressSanitizer causes the program to exit, but it could also be genuinely trying to run something that is not enabled on the current CPU.

If you do not need the sanitizers, as before you could remove the option, or at least remove Address:

   -DLLVM_USE_SANITIZER="Address;Undefined" \

We claim it is supported on "Linux" (https://clang.llvm.org/docs/AddressSanitizer.html#supported-platforms) but do not list architectures. I suppose as a software based sanitizer it would "just work"? But that's wishful thinking.

RISC-V support could be work in progress also, I would not know.

Or perhaps you are on a RISC-V board that does not include some hardware extensions? You could re-run that min-tblgen command under gdb and find out what that encoding is and decode it by hand.

If it's undefined on your hardware, you can try passing a more specific -march= setting to clang. Like before, you'd add this to -DCMAKE_C/CXX_FLAGS.

How you find the proper value for -march, I don't know. Maybe the board's manual has a recommendation?

If that doesn't work, it's possible some feature of the sanitizer requires that extension. You could look in llvm-project/compiler-rt for any references to it if you want to find out more.

[vijayaramaraju-kalidindi]

Hi @DavidSpickett

Hardware is not a board, it is QEMU emulated RISC Architecture VM on which compilation is being done.

Will retry compiling excluding this flag -DLLVM_USE_SANITIZER="Address;Undefined"

[DavidSpickett]

I seem to recall that AddressSanitizer does not like running in qemu-user simulation, but if you've got a whole system simulation with qemu-system that would in theory be fine.

So yes, I would expect QEMU to default to some "max" CPU with as many extensions as possible.

[vijayaramaraju-kalidindi]

After 25 hours of build time below is the result of regression tests.

Testing Time: 1294.87s

Total Discovered Tests: 3116
Unsupported : 226 (7.25%)
Passed : 2650 (85.04%)
Expectedly Failed: 1 (0.03%)
Failed : 239 (7.67%)
FAILED: tools/mlir/test/CMakeFiles/check-mlir /root/llvm-project-main/build/tools/mlir/test/CMakeFiles/check-mlir
cd /root/llvm-project-main/build/tools/mlir/test && /usr/local/bin/python3.10 /root/llvm-project-main/build/./bin/llvm-lit -sv /root/llvm-project-main/build/tools/mlir/test
ninja: build stopped: subcommand failed.

Can we know the cause for failed tests??

Also with 7% of failed cases can we move ahead using the MLIR compiler.

Have attached build logs.

MLIR Build Logs.txt

[MaskRay]

That said, we should consider whether the documentation should change to note this problem. Or give a command that would not lead to this problem.

@MaskRay since you wrote maskray.me/blog/2021-03-14-the-dark-side-of-riscv-linker-relaxation, do you know why these options are incompatible? Is it something where this only effects one linker or is it a generic problem?

Not fundamentally incompatible, but LLVM's debug info has supported this yet... Explained in this part of the blog post:

-gsplit-dwarf produces a .dwo file which will not be processed by the linker. If .dwo files contain relocations, they will not be resolved. Therefore the practice is that .dwo files do not contain relocations.

Currently Clang and GCC use DW_AT_high_pc or DW_AT_ranges to describe address ranges where the range sizes/endpoints are described by relocations. Such implementations are incompatible with linker relaxation. To work with linker relaxation, DW_AT_high_pc and DW_AT_ranges need to use indexes into .debug_addr (e.g. DW_RLE_startx_endx).

For now, either specify -mno-relax (slight text section increase, slightly faster compilation/linking) or disable split DWARF.
I'd recommend -mno-relax.

[vijayaramaraju-kalidindi]

Hi @DavidSpickett and @MaskRay,

Can we know the cause for failed tests??

Also with 7% of failed cases can we move ahead using the MLIR compiler.

[DavidSpickett]

Well one some level it is up to you to assess what level of quality you need to start your work. There will always be some issues on new platforms like RSIC-V.

In this specific case, I think there is one theme:

$ cat MLIR.Build.Logs.txt | grep "Unimplemented reloc type" | sort | uniq -c
      3 # |             1: LLVM ERROR: Unimplemented reloc type: 53
      2 # |             1: LLVM ERROR: Unimplemented reloc type: 54
      1 # |             1: LLVM ERROR: Unimplemented reloc type: 55
      1 # | LLVM ERROR: Unimplemented reloc type: 28
      9 # | LLVM ERROR: Unimplemented reloc type: 53
     57 # | LLVM ERROR: Unimplemented reloc type: 54
    139 # | LLVM ERROR: Unimplemented reloc type: 55

I believe these are all tests that try to JIT some code so I think there are some RISC-V relocation types it cannot handle.

You would need to find out what those relocation types are, and either find an option to not produce them or add support for them (I'm not sure in which component).