LLVM ERROR: Unsupported calling convention for call for llvm/clang 14.0.5

[littlewu2508]

I'm building rocm-comilersupport against llvm/clang 14.0.5. The build was successful, but many compile test failed:

The following tests FAILED:
         12 - comgr_compile_test (Subprocess aborted)
         13 - comgr_compile_minimal_test (Subprocess aborted)
         16 - comgr_compile_device_libs_test (Subprocess aborted)
         17 - comgr_compile_source_with_device_libs_to_bc_test (Subprocess aborted)
Errors while running CTest

They all fail with

LLVM ERROR: Unsupported calling convention for call

I backtraced and find out in https://github.com/llvm/llvm-project/blob/c12386ae247c0d46e1d513942e322e3a0510b126/llvm/lib/Target/AMDGPU/AMDGPUISelLowering.cpp#L980, CC is set to 91 (CallingConv::AMDGPU_KERNEL), while if I build comgr with ROCm's llvm, then the value is 0 (CallingConv::C).

I continoused tracing, until I lost track of this value at https://github.com/llvm/llvm-project/blob/c12386ae247c0d46e1d513942e322e3a0510b126/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp#L1136. The value hit 0 twice, and then hit 91 which causes unsupported convention error.

Also, running AMD_COMGR_SAVE_TEMPS=1 AMD_COMGR_REDIRECT_LOGS=stdout AMD_COMGR_EMIT_VERBOSE_LOGS=1 ./compile_minimal_test:

amd_comgr_do_action:
          ActionKind: AMD_COMGR_ACTION_COMPILE_SOURCE_TO_BC
             IsaName: amdgcn-amd-amdhsa--gfx803
             Options: "-mllvm" "-amdgpu-early-inline-all"
                Path:
            Language: AMD_COMGR_LANGUAGE_OPENCL_1_2
COMGR::executeInProcessDriver argv: clang "-cc1" "-triple" "amdgcn-amd-amdhsa" "-emit-llvm-bc" "-emit-llvm-uselists" "-clear-ast-before-backend" "-main-file-name" "source1.cl" "-mrelocation-model" "pic" "-pic-level" "1" "-fhalf-no-semantic-interposition" "-mframe-pointer=none" "-fdenormal-fp-math-f32=preserve-sign,preserve-sign" "-ffp-contract=on" "-fno-rounding-math" "-mconstructor-aliases" "-fvisibility" "hidden" "-fapply-global-visibility-to-externs" "-target-cpu" "gfx803" "-mllvm" "-treat-scalable-fixed-error-as-warning" "-debugger-tuning=gdb" "-resource-dir" "../../../../lib/clang/14.0.5" "-I" "/tmp/comgr-f8c1b9/include" "-isysroot" "/opt/gentoo" "-O3" "-std=cl1.2" "-fdebug-compilation-dir=/ext4-disk/build/portage/dev-libs/rocm-comgr-5.0.2-r1/work/rocm-comgr-5.0.2_build/test" "-ferror-limit" "19" "-fgnuc-version=4.2.1" "-fno-threadsafe-statics" "-fcolor-diagnostics" "-vectorize-loops" "-vectorize-slp" "-mllvm" "-amdgpu-early-inline-all" "-faddrsig" "-o" "/tmp/comgr-f8c1b9/output/source1.cl.bc" "-x" "cl" "/tmp/comgr-f8c1b9/input/source1.cl"
COMGR::executeInProcessDriver argv: clang "-cc1" "-triple" "amdgcn-amd-amdhsa" "-emit-llvm-bc" "-emit-llvm-uselists" "-clear-ast-before-backend" "-main-file-name" "source2.cl" "-mrelocation-model" "pic" "-pic-level" "1" "-fhalf-no-semantic-interposition" "-mframe-pointer=none" "-fdenormal-fp-math-f32=preserve-sign,preserve-sign" "-ffp-contract=on" "-fno-rounding-math" "-mconstructor-aliases" "-fvisibility" "hidden" "-fapply-global-visibility-to-externs" "-target-cpu" "gfx803" "-mllvm" "-treat-scalable-fixed-error-as-warning" "-debugger-tuning=gdb" "-resource-dir" "../../../../lib/clang/14.0.5" "-I" "/tmp/comgr-f8c1b9/include" "-isysroot" "/opt/gentoo" "-O3" "-std=cl1.2" "-fdebug-compilation-dir=/ext4-disk/build/portage/dev-libs/rocm-comgr-5.0.2-r1/work/rocm-comgr-5.0.2_build/test" "-ferror-limit" "19" "-fgnuc-version=4.2.1" "-fno-threadsafe-statics" "-fcolor-diagnostics" "-vectorize-loops" "-vectorize-slp" "-mllvm" "-amdgpu-early-inline-all" "-faddrsig" "-o" "/tmp/comgr-f8c1b9/output/source2.cl.bc" "-x" "cl" "/tmp/comgr-f8c1b9/input/source2.cl"
        ReturnStatus: AMD_COMGR_STATUS_SUCCESS

amd_comgr_do_action:
          ActionKind: AMD_COMGR_ACTION_LINK_BC_TO_BC
             IsaName: amdgcn-amd-amdhsa--gfx803
             Options: "-mllvm" "-amdgpu-early-inline-all"
                Path:
            Language: AMD_COMGR_LANGUAGE_OPENCL_1_2
        ReturnStatus: AMD_COMGR_STATUS_SUCCESS

amd_comgr_do_action:
          ActionKind: AMD_COMGR_ACTION_CODEGEN_BC_TO_RELOCATABLE
             IsaName: amdgcn-amd-amdhsa--gfx803
             Options: "-mllvm" "-amdgpu-early-inline-all"
                Path:
            Language: AMD_COMGR_LANGUAGE_OPENCL_1_2
COMGR::executeInProcessDriver argv: clang "-cc1" "-triple" "amdgcn-amd-amdhsa" "-emit-obj" "-mrelax-all" "--mrelax-relocations" "-clear-ast-before-backend" "-main-file-name" "linked.bc" "-mrelocation-model" "pic" "-pic-level" "1" "-fhalf-no-semantic-interposition" "-mframe-pointer=all" "-fdenormal-fp-math-f32=preserve-sign,preserve-sign" "-ffp-contract=on" "-fno-rounding-math" "-mconstructor-aliases" "-fvisibility" "hidden" "-fapply-global-visibility-to-externs" "-target-cpu" "gfx803" "-mllvm" "-treat-scalable-fixed-error-as-warning" "-debugger-tuning=gdb" "-resource-dir" "../../../../lib/clang/14.0.5" "-fdebug-compilation-dir=/ext4-disk/build/portage/dev-libs/rocm-comgr-5.0.2-r1/work/rocm-comgr-5.0.2_build/test" "-ferror-limit" "19" "-fgnuc-version=4.2.1" "-fcolor-diagnostics" "-mllvm" "-amdgpu-early-inline-all" "-faddrsig" "-o" "/tmp/comgr-da2c2e/output/linked.bc.o" "-x" "ir" "/tmp/comgr-da2c2e/input/linked.bc"
LLVM ERROR: Unsupported calling convention for call
zsh: abort      AMD_COMGR_SAVE_TEMPS=1 AMD_COMGR_REDIRECT_LOGS=stdout =1

Directly running from cmdline also suffers:

clang "-cc1" "-triple" "amdgcn-amd-amdhsa" "-emit-obj" "-mrelax-all" "--mrelax-relocations" "-clear-ast-before-backend" "-main-file-name" "linked.bc" "-mrelocation-model" "pic" "-pic-level" "1" "-fhalf-no-semantic-interposition" "-mframe-pointer=all" "-fdenormal-fp-math-f32=preserve-sign,preserve-sign" "-ffp-contract=on" "-fno-rounding-math" "-mconstructor-aliases" "-fvisibility" "hidden" "-fapply-global-visibility-to-externs" "-target-cpu" "gfx803" "-mllvm" "-treat-scalable-fixed-error-as-warning" "-debugger-tuning=gdb" "-resource-dir" "../../../../lib/clang/14.0.5" "-fdebug-compilation-dir=/ext4-disk/build/portage/dev-libs/rocm-comgr-5.0.2-r1/work/rocm-comgr-5.0.2_build/test" "-ferror-limit" "19" "-fgnuc-version=4.2.1" "-fcolor-diagnostics" "-mllvm" "-amdgpu-early-inline-all" "-faddrsig" "-o" "/tmp/comgr-da2c2e/output/linked.bc.o" "-x" "ir" "/tmp/comgr-da2c2e/input/linked.bc"
fatal error: error in backend: Unsupported calling convention for call

[littlewu2508]

Also to mention, due to llvm's change https://reviews.llvm.org/D116011, comgr should be patched:

diff --git a/lib/comgr/src/comgr-compiler.cpp b/lib/comgr/src/comgr-compiler.cpp
index 465187e..181cb81 100644
--- a/lib/comgr/src/comgr-compiler.cpp
+++ b/lib/comgr/src/comgr-compiler.cpp
@@ -666,6 +666,7 @@ AMDGPUCompiler::executeInProcessDriver(ArrayRef<const char *> Args) {
     initializeCommandLineArgs(Argv);
     Argv.append(Arguments.begin(), Arguments.end());
     Argv.push_back(nullptr);
+    Argv.pop_back();

     // By default clang driver will ask CC1 to leak memory.
     auto *IT = find(Argv, StringRef("-disable-free"));

Other wise test will encounter

terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
zsh: abort      ./compile_minimal_test

[littlewu2508]

Confirm that Fedora and Debian both encountering this problem. Now I'm trying to package rocm-comgr against llvm-14 on Gentoo and find the same issue exists.

[lamb-j]

Hi @littlewu2508, thanks for looking into this!

Regarding the second issue you mentioned (errors due to llvm's change https://reviews.llvm.org/D116011), can you verify which version of Comgr you're testing with and see if it includes a patch we added related to this (ROCm/ROCm-CompilerSupport@a75326c). This should cover your "cc1" case, but if you're still experiencing errors with this patch we may need to revisit.

I'm still looking into the "LLVM ERROR: Unsupported calling convention for call" error you first mentioned, and will update here when I find out anything.

[preda]

I see the same (LLVM ERROR: Unsupported calling convention for call) with LLVM main branch.

[littlewu2508]

Hi @littlewu2508, thanks for looking into this!

Regarding the second issue you mentioned (errors due to llvm's change https://reviews.llvm.org/D116011), can you verify which version of Comgr you're testing with and see if it includes a patch we added related to this (a75326c). This should cover your "cc1" case, but if you're still experiencing errors with this patch we may need to revisit.

Previously I'm using comgr-5.0.2. Now with 5.1.x and 5.2 the bug is gone. Thanks for pointing out that you already have the fix.

[littlewu2508]

I see the same (LLVM ERROR: Unsupported calling convention for call) with LLVM main branch.

Hello, I'd like to know is there any progress on this?

[MathiasMagnus]

I too am hitting this issue.

I'm working on a project that uses comgr as a library to process LLVM bitcode coming from elsewhere. Incoming LLVM bitcode may be of mostly any LLVM version (that I choose when building that part of my program), and then I feed that bitcode comgr and would want to use it to build/link/etc in terms of comgr. When adding a device library and trying to link the resulting dataset, I too get LLVM ERROR: Unsupported calling convention for call. I built a debug version of LLVM to see what's happening and can confirm the same enum value (91) of calling convention, but I do not know which node it comes or happens on. (I'm a total LLVM newbie.)

Can anyone point me toward working release versions of LLVM and comgr that are known to work? Possibly as recent as reasonably possible.

[littlewu2508]

I too am hitting this issue.

I'm working on a project that uses comgr as a library to process LLVM bitcode coming from elsewhere. Incoming LLVM bitcode may be of mostly any LLVM version (that I choose when building that part of my program), and then I feed that bitcode comgr and would want to use it to build/link/etc in terms of comgr. When adding a device library and trying to link the resulting dataset, I too get LLVM ERROR: Unsupported calling convention for call. I built a debug version of LLVM to see what's happening and can confirm the same enum value (91) of calling convention, but I do not know which node it comes or happens on. (I'm a total LLVM newbie.)

Can anyone point me toward working release versions of LLVM and comgr that are known to work? Possibly as recent as reasonably possible.

The AMD forked LLVM works without issue: https://github.com/RadeonOpenCompute/llvm-project/

I also built a debug version but still cannot understand the reason. I tried to bisect it from the merge-base of vanilla llvm but there are too many build failures. Hope you can find some clue!

[MathiasMagnus]

@littlewu2508 Which commit hash are you using to build the AMD fork of llvm-project and comgr? I still can't run examples like compile_minimal_test.exe without this error. Also, which ROCm device library version are you using that makes find_package(AMDDeviceLibs REQUIRED CONFIG) succeed?

[littlewu2508]

@littlewu2508 Which commit hash are you using to build the AMD fork of llvm-project and comgr? I still can't run examples like compile_minimal_test.exe without this error. Also, which ROCm device library version are you using that makes find_package(AMDDeviceLibs REQUIRED CONFIG) succeed?

As I remember I was using ROCm version 5.1.3, whose amd-llvm's commit hash is 5cba46f, and comgr commit hash is 11321060a69dc41f8bed6f6d8f900ad4c146025b.

When I tried bisecting one month ago, I use ROCm-5.4.0 because it's merge-base with vanilla llvm is near llvm-15 release. In that case amd-llvm's commit hash is d6f0fe8. All amd-llvm can make comgr-5.1.3 pass compile_minimal_test.

For AMDDeviceLibs I use ROCm version 5.1.3

[littlewu2508]

Confirmed that this issue still exists for llvm-16 (llvm commit dafebd5b5a08dde25f5f52f65cac54bd6ec0ecde) and comgr development branch: (comgr commit 4c092fc02e59a2783c94a23a89d2987a2f5a5239)

[littlewu2508]

Some progress:

I managed to reproduce using pure clang commands. The reproducing method is described in https://github.com/littlewu2508/LLVMAMDGPUcodegenbug. The gdb trace follows similar steps of compile_minimal_test, with less steps going to the Instruction I having Opcode 56 (CallConv) with SubclassData ((365>>2 = 91 which is CallingConv::AMDGPU_KERNEL, I guess) that causes this issue.

Since without comgr I can also reproduce this I will report that to llvm team, too

[littlewu2508]

I too am hitting this issue.

I'm working on a project that uses comgr as a library to process LLVM bitcode coming from elsewhere. Incoming LLVM bitcode may be of mostly any LLVM version (that I choose when building that part of my program), and then I feed that bitcode comgr and would want to use it to build/link/etc in terms of comgr. When adding a device library and trying to link the resulting dataset, I too get LLVM ERROR: Unsupported calling convention for call. I built a debug version of LLVM to see what's happening and can confirm the same enum value (91) of calling convention, but I do not know which node it comes or happens on. (I'm a total LLVM newbie.)

It is now clear that this issue is not specific to comgr. It's because upstream llvm does not support handling calls to kernels.

Can you give an example of your bitcode and the source to generate them?