Please consider installing `std.cppm` in `lib++`

[boris-kolpackov]

I've been trying out the std module in libc++ with build2 (per the Modules in libc++ documentation) and everything seems to work without any issues, at least on simple examples. Very impressive, I must say.

There is just one snag: std.cppm is not installed. Quoting the relevant part from the above documentation:

Once libc++'s implementation is more mature we will reach out to build system vendors, with the goal that building the BMI files is done by the build system.

Currently this requires a local build of libc++ with modules enabled. Since modules are not part of the installation yet, they are used from the build directory.

In build2 we have the ability to build BMIs of external libraries on the fly (this is needed for consuming modules from installed libraries). So if std.cppm were installed somewhere where we could find it, then this would work pretty much out of the box in our case.

I can see two immediate questions about installing std.cppm: where to install it and how to make it self-contained (currently, it #includes a large number of files from the std/ subdirectory).

Regarding where to install, I would suggest just installing it next to the headers (e.g., into /usr/include/c++/v1/). I can think of two advantage with doing it this way:

1. There is already a way for a build system to find this directory (by parsing the header search paths in the -v output) and some build systems already do this (for example, in build2 we extract the compiler's header and library search paths).

2. The std.cppm file will end up in the right distribution package (e.g., -dev on Debian, -devel on Fedora, etc) without any extra effort.

The only objection I've heard to installing modules next to headers is that "it's not a header" and "it may be included by mistake". While both are true, we are already installing non-header files (e.g., inline/template implementation files) and nobody seems to be including them by mistake. We could also take an extra measure to preclude this by, for example, installing the modules into a subdirectory (e.g., /usr/include/c++/v1/modules/).

Regarding making std.cppm self-contained, I assume there is no desire to install all the 100+ files from std/ that it includes (though there is nothing technically bad about doing it, just feels unnecessary). It seems the easiest would be to just append their contents to std.cppm.in instead of #includeing them. Though I am not sure if this is easy to achieve in CMake.

Thoughts?

P.S. For completeness, let me also mention what I've done as a (hopefully) temporary measure in build2 to support import std right now. I've made a self-contained std.cppm (as described above), patched it slightly so that it works for both libc++ 17 and 18, and bundled it with build2. In other words, instead of being installed with libc++, it is being installed with build2 (again, hopefully temporarily and into build2's own directory, naturally).

@mordante @iains @ChuanqiXu9 @dwblaikie @mathstuf

[llvmbot]

@llvm/issue-subscribers-clang-modules

Author: Boris Kolpackov (boris-kolpackov)

I've been trying out the `std` module in `libc++` with `build2` (per the [Modules in libc++](https://github.com/llvm/llvm-project/blob/main/libcxx/docs/Modules.rst) documentation) and everything seems to work without any issues, at least on simple examples. Very impressive, I must say.

There is just one snag: std.cppm is not installed. Quoting the relevant part from the above documentation:

> Once libc++'s implementation is more mature we will reach out to build system vendors, with the goal that building the BMI files is done by the build system.
>
> Currently this requires a local build of libc++ with modules enabled. Since modules are not part of the installation yet, they are used from the build directory.

In build2 we have the ability to build BMIs of external libraries on the fly (this is needed for consuming modules from installed libraries). So if std.cppm were installed somewhere where we could find it, then this would work pretty much out of the box in our case.

I can see two immediate questions about installing std.cppm: where to install it and how to make it self-contained (currently, it #includes a large number of files from the std/ subdirectory).

Regarding where to install, I would suggest just installing it next to the headers (e.g., into /usr/include/c++/v1/). I can think of two advantage with doing it this way:

1. There is already a way for a build system to find this directory (by parsing the header search paths in the -v output) and some build systems already do this (for example, in build2 we extract the compiler's header and library search paths).

2. The std.cppm file will end up in the right distribution package (e.g., -dev on Debian, -devel on Fedora, etc) without any extra effort.

The only objection I've heard to installing modules next to headers is that "it's not a header" and "it may be included by mistake". While both are true, we are already installing non-header files (e.g., inline/template implementation files) and nobody seems to be including them by mistake. We could also take an extra measure to preclude this by, for example, installing the modules into a subdirectory (e.g., /usr/include/c++/v1/modules/).

Regarding making std.cppm self-contained, I assume there is no desire to install all the 100+ files from std/ that it includes (though there is nothing technically bad about doing it, just feels unnecessary). It seems the easiest would be to just append their contents to std.cppm.in instead of #includeing them. Though I am not sure if this is easy to achieve in CMake.

Thoughts?

P.S. For completeness, let me also mention what I've done as a (hopefully) temporary measure in build2 to support import std right now. I've made a self-contained std.cppm (as described above), patched it slightly so that it works for both libc++ 17 and 18, and bundled it with build2. In other words, instead of being installed with libc++, it is being installed with build2 (again, hopefully temporarily and into build2's own directory, naturally).

@mordante @iains @ChuanqiXu9 @dwblaikie @mathstuf

[ChuanqiXu9]

FYI, about the position of installing modules, there is a discussion in https://lists.isocpp.org/sg15/2023/07/1971.php

[boris-kolpackov]

BTW, to add a data point for the "perhaps the std module in libc++ is mature enough to start installing it" position: I was able to build @ChuanqiXu9 's simple_async CXX20Modules branch with libc++. All I had to do is fix a few bugs in the project and get rid of one workaround (which is not standard C++). You can see the changes here: https://github.com/boris-kolpackov/async_simple/tree/CXX20Modules-build2

[mathstuf]

Note that we don't just need the files, but also the flags to make them work as expected (namely to suppress warnings about reserved module names via -Wno-reserved-module-identifier).

[ldionne]

If we start shipping std.cppm at a given location, that will become the de-facto standard for build systems to pick up modules files from. I'm not really opposed to doing that but it seems like we'd be stepping on SG15's toes since my understanding is that we haven't come to an official agreement on that yet?

CC @Bigcheese

[mordante]

Great to hear it works with build2!

Shipping the module files is getting at the top of my todo list and I wanted to investigate which tool vendors to contact. However last week I saw @ruoso created cplusplus/modules-ecosystem-tr#29. So I prefer to participate in that effort instead of starting a similar effort. Build2 is also on that list.

Regarding the large amount of inc files. For now I intend to keep them, they are very useful for testing that libc++ exports all named declarations from a header.

[boris-kolpackov]

Note that we don't just need the files, but also the flags to make them work as expected (namely to suppress warnings about reserved module names via -Wno-reserved-module-identifier).

FWIW, this is the only extra option I had to pass to compile std.cppm. At least in build2 this whole logic is quite ad hoc: detect the importation of the std modules, notice that it's clang with libc++, find std.cppm, synthesize the dependency for the BMI, and add the extra option. And I can't see any major generalization in this area. Feels like build systems will just need to handle each compiler/standard library combination. Luckily there aren't that many of them.

I'm not really opposed to doing that but it seems like we'd be stepping on SG15's toes since my understanding is that we haven't come to an official agreement on that yet?

How can I put this politely? SG15 still hasn't come to an official agreement on the file extension for module but it seems to have agreed on the term used for compiled modules: BMI. Though there is no agreement on what it stands for: some think it's a "binary module interface", some think it's a "built module interface", there is probably another interpretation I am not aware of. So if you wait on SG15, you may end up in a situation where build systems just run out of patience and start bundling your module interfaces.

Also, FWIW, MSVC folks have decided on the location and are guaranteeing it: build2/build2#333 (comment)

Regarding the large amount of inc files. For now I intend to keep them, they are very useful for testing that libc++ exports all named declarations from a header.

Just to clarify, I didn't suggest that you get rid of them entirely, only in the installed version of std.cppm. Even that was only a suggestions -- if you decide to install them, I don't think anyone will complain.

[mathstuf]

Feels like build systems will just need to handle each compiler/standard library combination. Luckily there aren't that many of them.

There are at least a half dozen clang-based frontends with different options and no telling module flags will be one such disagreement point or not (hopefully, but some have different flag patterns they may wish to follow). It don't think it is so trivial unless one only wants to support upstream clang, Apple clang, gcc, and MSVC.

[ChuanqiXu9]

From cplusplus/modules-ecosystem-tr#29:

The first step of this process will be to collect input from a wide range of tooling providers in order to identify requirements in the ecosystem, for that we need to identify who we need to reach out to in order to get the requirements.

So it looks like that issue is not a good place to talk about the installation place for std modules.

If we want to push things forward, maybe we can send another mail to SG15 with a stronger desire to get a conclusion (instead of purely discussion), or if we want, we can make the decision ourselves. (I don't have a strong opinion here)

CC: @mordante @ldionne @boris-kolpackov