[clang, CUDA, modules] Issues Compiling a CUDA Source File with C++ Modules

[LukasBreitwieser]

Hi All,

I ran into issues compiling a simple CUDA source file that includes a header (see mymodule.h) if modules are enabled. Further investigation showed that the issue persists even after removing all CUDA related code (see main.cu). Compiling it with c++20 errors out due to an ambiguous call to operator new, while the compiler crashes with c++17. Here the compilation logs and generated *.pcm files: logs-and-module-cache.tar.gz

File main.cu

#include "mymodule.h"
#include <iostream>

int main() {
  std::cout << GetMessage() << std::endl;
  // Problem shows up even with all CUDA related code removed.
  return 0;
}

File mymodule.h

#ifndef MYMODULE_H
#define MYMODULE_H
#include <string>
// uncommenting the following line somehow "fixes" the issue.
//#include <iostream> 

inline std::string GetMessage() { 
  return "Hello world!";
}

#endif // MYMODULE_H

File module.modulemap

module mymodule {
  header "mymodule.h"
  export *
}

Compiler invocation:

clang++ \
  -v \
  -std=c++20 \
  -x cuda \
  --cuda-gpu-arch=sm_86 \
  -lcuda \
  -lcudart \
  -fmodules \
  -fmodule-map-file=module.modulemap \
  -fmodules-cache-path=/tmp/clang-modules \
  -o main \
  main.cu

LLVM commit

commit 2b55ef187cb6029eed43d7f4c0a3640c32691b31 (HEAD -> main, origin/main, origin/HEAD)
Author: Florian Hahn <flo@fhahn.com>
Date:   Wed Jan 29 10:50:01 2025 +0000

    [VPlan] Add helper to run VPlan passes, verify after run (NFC). (#123640)
...

LLVM configuration

cmake -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
    -DLLVM_ENABLE_PROJECTS="clang" \
    -DLLVM_TARGETS_TO_BUILD="host;NVPTX" \
    <path-to-llvm-project-dir>/llvm

System Info

• OS: Ubuntu 20.04.1 LTS
• Compiler used to compile llvm: g++ (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
• CUDA: 12.6
• GPU: single NVIDIA Geforce RTX 3060 with compute capability 8.6

I am happy to provide more information if needed.
Thank you in advance for taking a look and any hints on how to best debug and fix it.

Best,
Lukas

[llvmbot]

@llvm/issue-subscribers-clang-modules

Author: Lukas Breitwieser (LukasBreitwieser)

Hi All,

I ran into issues compiling a simple CUDA source file that includes a header (see mymodule.h) if modules are enabled. Further investigation showed that the issue persists even after removing all CUDA related code (see main.cu). Compiling it with c++20 errors out due to an ambiguous call to operator new, while the compiler crashes with c++17. Here the compilation logs and generated *.pcm files: logs-and-module-cache.tar.gz

File main.cu

#include "mymodule.h"
#include <iostream>

int main() {
  std::cout << GetMessage() << std::endl;
  // Problem shows up even with all CUDA related code removed.
  return 0;
}

File mymodule.h

#ifndef MYMODULE_H
#define MYMODULE_H
#include <string>
// uncommenting the following line somehow "fixes" the issue.
//#include <iostream> 

inline std::string GetMessage() { 
  return "Hello world!";
}

#endif // MYMODULE_H

File module.modulemap

module mymodule {
  header "mymodule.h"
  export *
}

Compiler invocation:

clang++ \
  -v \
  -std=c++20 \
  -x cuda \
  --cuda-gpu-arch=sm_86 \
  -lcuda \
  -lcudart \
  -fmodules \
  -fmodule-map-file=module.modulemap \
  -fmodules-cache-path=/tmp/clang-modules \
  -o main \
  main.cu

LLVM commit

commit 2b55ef187cb6029eed43d7f4c0a3640c32691b31 (HEAD -> main, origin/main, origin/HEAD)
Author: Florian Hahn &lt;flo@<!-- -->fhahn.com&gt;
Date:   Wed Jan 29 10:50:01 2025 +0000

    [VPlan] Add helper to run VPlan passes, verify after run (NFC). (#<!-- -->123640)
...

LLVM configuration

cmake -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
    -DLLVM_ENABLE_PROJECTS="clang" \
    -DLLVM_TARGETS_TO_BUILD="host;NVPTX" \
    <path-to-llvm-project-dir>/llvm

System Info

• OS: Ubuntu 20.04.1 LTS
• Compiler used to compile llvm: g++ (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
• CUDA: 12.6
• GPU: single NVIDIA Geforce RTX 3060 with compute capability 8.6

I am happy to provide more information if needed.
Thank you in advance for taking a look and any hints on how to best debug and fix it.

Best,
Lukas

[Artem-B]

Caveat -- I'm not familiar specifically with modules, and assume that they work roughly the same as precompiled headers.

At the moment precompiled headers do not work with CUDA compilation, though you may be able to use it in some limited circumstances.

The fundamental issue is that PCH (and I assume PCM) files require exactly the same compiler and compilation flags to be used by both the creator and the user of the file. CUDA compilation spawns N sub-compilations, so any given PCH file would only be suitable for one of those sub-compilations.

That assumes that you even manage to create a valid PCM/PCH file. Last time I tried, each sub-compilation ended up trying to create that PCH file, and the last one (host) won. When passed to subsequent CUDA compilation that PCH file was rejected because it was created by and intended for the host compilation.

Another issue is/was that deserialization of AST didn't take into account that CUDA has target-attribute based function overloading, and treated all functions with host or device attributes as redeclarations.
#125127 fixed some of these issues, but there may be other problems. CUDA does have a lot of special cases normally done by Sema, and AST deserialization may still be missing some CUDA-specific quirks.

So, the bottom line is that in practice PCH/PCM are likely only usable for the host side of the CUDA compilation and there's no convenient way (yet) to have PCM/PCH enabled for the host side compilation only. Considering that most of the compilation time is spent on the GPU compilations, the benefit of PCM/PCH use for the host compilation only would be rather small, anyways.

[aaronmondal]

See also #57862.

While it's not "ideal", it's still possible to mix (the standard modules, not the modulemap) modules with gpu code in a codebase by wrapping module code in headers and using classic includes for device code: https://github.com/eomii/rules_ll/tree/main/examples/hip_modules_example

[LukasBreitwieser]

Thank you very much for your replies!

Is module support for CUDA compilations currently being worked on or on the roadmap?
If not, how much development time do you think is necessary to make the necessary changes?

A bit of background. I am working on enabling CUDA support for the data analysis platform ROOT with >2M lines of C++ code. ROOT uses the C++ interpreter cling (similar to clang-repl) to enable interactive physics data analyses and relies on modulemap modules to rapidly load the provided headers. As of now, the amount of CUDA code will be minimal compared to the host side. Therefore, I would need at least support for modulemap modules for the host.

I tried to get the reproducer to work only on the host side (by adding --cuda-host-only) but ran into the same error messages from my initial post.

Many thanks in advance for your help!

[Artem-B]

Is module support for CUDA compilations currently being worked on or on the roadmap?

Not to my knowledge.

If not, how much development time do you think is necessary to make the necessary changes?

We've got most of the necessary pieces of the puzzle.
The missing bit is support of N different subcompilations in a single PCH file. The easiest way is to get "normal" PCH for each sub-compilation (let's call them singleton PCH) and pack them into some sort of container (multi-PCH?)

• create singleton PCH for each sub-compilation
• pack all singleton PCH from all sub-compilations into a single file, along with the properties of corresponding subcompilation
• during each compilation which uses PCH, select the matching singleton PCH from a multi-PCH container.

So, my guess is that it should be doable with a moderate amount of work to implement multi-PCH file format, and then plumb multi-PCH creation and singleton-PCH selection machinery into clang driver. The PCH compilation mechanics itself should remain largely unchanged.