Libcxx's document of C++ std library modules is outdated

[a858438680]

Libcxx's document of C++ std library modules in Modules in libc++ is outdated, which is compiled from Modules.rst.

In this document, the std library modules are downloaded from build directory into project directory through FetchContent. However, this requires the downloaded library to be a CMake project, which does not hold anymore after the #76083 is merged. Now people following this document will find it confusing and frustrating.

Fortunately, the author provides another way to use the std modules in #75741 , which uses the CMake option -DLIBCXX_INSTALL_MODULES=ON to install the std.cppm and std.compat.cppm. However the author said that currently there is not build system support. But I have explored a premature way to use the installed std modules in CMake. I'd like to contribute it to the libcxx's document. But I'm not sure whether such a pull request is acceptable.

[mordante]

I (the main author of that page) would certainly be happy to have documentation how to use installed modules without build system support, thanks for offering to write it!
The end-goal is indeed to have build system support. Since installation is optional we need to keep the existing documentation. I would prefer to have the new documentation before the existing documentation with the downloaded build.

[a858438680]

I (the main author of that page) would certainly be happy to have documentation how to use installed modules without build system support, thanks for offering to write it! The end-goal is indeed to have build system support. Since installation is optional we need to keep the existing documentation. I would prefer to have the new documentation before the existing documentation with the downloaded build.

Thanks for your reply. Before writing the document, I'd like to ask some questions about the build of llvm and clang. I successfully configured and built llvm 18.1.0-rc1 using the source code in Github release page. However, I think that the document in the main branch should use source code from git, right? But I can not build llvm-git using basically the same configuration.

The build command I used for release source code is

mv clang llvm/tools/clang
mv clang-tools-extra llvm/tools/clang/tools/extra
mv lld llvm/tools/lld
mv lldb llvm/tools/lldb
cp cmake/Modules/* llvm/cmake/modules/

mkdir build

cmake -S llvm \
    -B build \
    -G "Unix Makefiles" \
    -DCMAKE_BUILD_TYPE=Release \
    -DLLVM_ENABLE_RUNTIMES="all" \
    -DLLVM_INCLUDE_TESTS=OFF \
    -DLLVM_INCLUDE_BENCHMARKS=OFF \
    -DLIBCXX_INSTALL_MODULES=ON

cmake --build build -- -j 32

and the command to build llvm-git is

git clone https://github.com/llvm/llvm-project.git
cd llvm-project
cmake -S llvm \
    -B build \
    -G "Ninja" \
    -DCMAKE_BUILD_TYPE=Release \
    -DLLVM_ENABLE_RUNTIMES="all" \
    -DLIBCXX_INSTALL_MODULES=ON
cmake --build build -- -j32

Do you think it's ok to use a specific version of llvm in the document? Or could you give me some suggestions about how to build llvm-git?

[a858438680]

I (the main author of that page) would certainly be happy to have documentation how to use installed modules without build system support, thanks for offering to write it! The end-goal is indeed to have build system support. Since installation is optional we need to keep the existing documentation. I would prefer to have the new documentation before the existing documentation with the downloaded build.

And I'am also a little confused about what you mean to keep the existing documentation. Yes, the installation is optional but the CMakeLists.txt generated in the <build>/modules/v1/c++ is deleted unconditionally, which is required by the FetchContent method in the existing document. Whether the cppms are installed or not does not change the fact that the existing document is broken. Unless the remove of the CMakeLists.txt is done unintentionally.

To modify the document as possible as you may want it to be, I would prefer if you could explain it to me more in detail.

[a858438680]

@mordante Here, I have written a preliminary version of the new document. However, I think it still needs some modifications. You can check it and give me some feedbacks. After listening to your main thoughts, I will create a PR to change the document.

Requirements of build tools:

llvm and clang: 18.1.0-rc1 built with -DLIBCXX_INSTALL_MODULES=ON
cmake: 3.28
ninja: 1.11

Currently this requires a local build of llvm and clang with std modules installed. To build the llvm and clang, use the following scripts:

#!/bin/bash

VERSION=18.1.0-rc1

components=(
    "bolt"
    "clang"
    "clang-tools-extra"
    "cmake"
    "compiler-rt"
    "flang"
    "libclc"
    "libcxx"
    "libcxxabi"
    "libunwind"
    "lld"
    "lldb"
    "llvm"
    "mlir"
    "openmp"
    "polly"
    "runtimes"
    "test-suite"
    "third-party"
)

declare -A URLS
declare -A SRC_ROOTS
declare -A DIRS

for component in "${components[@]}"; do
    URLS[$component]="https://github.com/llvm/llvm-project/releases/download/llvmorg-${VERSION}/${component}-${VERSION/-/}.src.tar.xz"
    SRC_ROOTS[$component]="${component}-${VERSION/-/}.src"
    DIRS[$component]="${component}"
done

for component in "${components[@]}"; do
    if [ -d ${SRC_ROOTS[$component]} ]; then
        rm -rf ${SRC_ROOTS[$component]}
    fi
    if [ -d ${DIRS[$component]} ]; then
        rm -rf ${DIRS[$component]}
    fi
done

if [ -d "build" ]; then
    rm -rf build
fi

for component in "${components[@]}"; do
    if [ ! -f ${SRC_ROOTS[$component]}.tar.xz ]; then
        echo "Downloading ${SRC_ROOTS[$component]}.tar.xz"
        wget -O ${SRC_ROOTS[$component]}.tar.xz ${URLS[$component]}
    else
        echo "Found ${SRC_ROOTS[$component]}.tar.xz"
    fi
done

for component in "${components[@]}"; do
    echo "Extracting ${SRC_ROOTS[$component]}.tar.xz"
    tar xf ${SRC_ROOTS[$component]}.tar.xz
    mv ${SRC_ROOTS[$component]} ${DIRS[$component]}
done

mv clang llvm/tools/clang
mv clang-tools-extra llvm/tools/clang/tools/extra
mv lld llvm/tools/lld
mv lldb llvm/tools/lldb
cp cmake/Modules/* llvm/cmake/modules/

mkdir build

cmake -S llvm \
    -B build \
    -G "Unix Makefiles" \
    -DCMAKE_BUILD_TYPE=Release \
    -DLLVM_ENABLE_RUNTIMES="all" \
    -DLLVM_INCLUDE_TESTS=OFF \
    -DLLVM_INCLUDE_BENCHMARKS=OFF \
    -DLIBCXX_INSTALL_MODULES=ON

cmake --build build -- -j 32

After the llvm is built, you can install it to the path you like. For example, you can use cmake option -DCMAKE_INSTALL_PREFIX=/path/to/install during configuration or use DESTDIR=/path/to/install make install during installation. We refer to such /path/to/install as <install_prefix>.

Then we create an external project with root directory <project_root>. First we create <project_root>/CMakeLists.txt:

cmake_minimum_required(VERSION 3.28.0 FATAL_ERROR)

project(HelloWorld LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 23)
set(CMAKE_CXX_STANDARD_REQUIRED YES)
set(CMAKE_CXX_EXTENSIONS OFF)

include(std.cmake)

add_executable(main)

target_sources(main
  PRIVATE
    main.cpp
)

Then the <project_root>/std.cmake:

if (NOT CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
  message(FATAL_ERROR "std module requires Clang")
endif()

if (NOT CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL "18.1.0")
  message(FATAL_ERROR "std module requires Clang 18.1.0 or later")
endif()

execute_process(COMMAND bash "-c" "${CMAKE_CXX_COMPILER} -v 2>&1 > /dev/null | grep 'InstalledDir' | awk '{print $2}' | tr -d '\n'" OUTPUT_VARIABLE CLANG_DIR)
execute_process(COMMAND bash "-c" "${CMAKE_CXX_COMPILER} -v 2>&1 > /dev/null | grep 'Target: ' | awk '{print $2}' | tr -d '\n'" OUTPUT_VARIABLE CLANG_TARGET)

include(FetchContent)
FetchContent_Declare(
  std_module
  URL "file://${CLANG_DIR}/../share/libc++/v1"
  DOWNLOAD_EXTRACT_TIMESTAMP TRUE
  SYSTEM
)

if (NOT std_module_POPULATED)
  FetchContent_Populate(std_module)
endif()

add_library(std)
target_sources(std
  PUBLIC FILE_SET cxx_modules TYPE CXX_MODULES FILES
    ${std_module_SOURCE_DIR}/std.cppm
    ${std_module_SOURCE_DIR}/std.compat.cppm
)

target_compile_options(std
  PUBLIC
    -stdlib=libc++
    -Wno-reserved-module-identifier
    -Wno-reserved-user-defined-literal
)

target_link_options(std
  PUBLIC
    -stdlib=libc++
    -Wl,-rpath=${CLANG_DIR}/../lib/${CLANG_TARGET}
)

target_link_libraries(std
  INTERFACE
    c++
)

link_libraries(std)

Finally, we create the <project_root>/main.cpp

import std;

auto main() -> int {
    std::println("Hello, world!");
}

Then we can configure and build this external project:

mkdir build
cmake -S <project_root> -B build -G Ninja -DCMAKE_CXX_COMPILER=<install_prefix>/usr/local/bin/clang-18
cmake --build build
./build/main

If it goes well, now you can see:

Hello, world!

[mordante]

I (the main author of that page) would certainly be happy to have documentation how to use installed modules without build system support, thanks for offering to write it! The end-goal is indeed to have build system support. Since installation is optional we need to keep the existing documentation. I would prefer to have the new documentation before the existing documentation with the downloaded build.

And I'am also a little confused about what you mean to keep the existing documentation. Yes, the installation is optional but the CMakeLists.txt generated in the <build>/modules/v1/c++ is deleted unconditionally, which is required by the FetchContent method in the existing document. Whether the cppms are installed or not does not change the fact that the existing document is broken. Unless the remove of the CMakeLists.txt is done unintentionally.

Nice catch, that removal is indeed unintentionally. I'll re-add the CMakeLists.txt to the build directory. This file will not be installed. (It will be removed once installation is ON by default and build system support is available.)

I would prefer the documentation to only build the required parts. Your example builds the entire LLVM project and its subprojects. This is not required. Can you make a pull request? That makes it a lot easier to test it locally and comment on your changes.

[a858438680]

I (the main author of that page) would certainly be happy to have documentation how to use installed modules without build system support, thanks for offering to write it! The end-goal is indeed to have build system support. Since installation is optional we need to keep the existing documentation. I would prefer to have the new documentation before the existing documentation with the downloaded build.

And I'am also a little confused about what you mean to keep the existing documentation. Yes, the installation is optional but the CMakeLists.txt generated in the <build>/modules/v1/c++ is deleted unconditionally, which is required by the FetchContent method in the existing document. Whether the cppms are installed or not does not change the fact that the existing document is broken. Unless the remove of the CMakeLists.txt is done unintentionally.

Nice catch, that removal is indeed unintentionally. I'll re-add the CMakeLists.txt to the build directory. This file will not be installed. (It will be removed once installation is ON by default and build system support is available.)

I would prefer the documentation to only build the required parts. Your example builds the entire LLVM project and its subprojects. This is not required. Can you make a pull request? That makes it a lot easier to test it locally and comment on your changes.

Ok, actaully if the CMakeLists.txt will come back, then the document works again and I think this issue's main concern is solved.

However, I am still glad to contribute to the document. But my method acctually assumes that the libcxx is installed under the <Clang_InstalledDir>/../lib/<Clang_Target>, which does not require a configure time cmake option like LIBCXX_BUILD. If the runtimes are built and installed alone, I think such an option will still be required. Of course, if you think this way is better, I think I can figure out how to use std modules in this way.

So actually I have not tested how to use the sperately installed libcxx. At least I have to try it locally first. And then write another different document.

And I do not actually understand what you mean making a pull request. Because all my current method is here in my comment above. And there is nothing to change in the source code of llvm. All of these files are scripts and cmake file in the external project. So first, I can not make pull request of llvm's patch. And also, I think you are not satisfied with my method now. So second, I can not make a pull request of document. Finally, in general, I don't understand what kind of pull request you want me to make.

[a858438680]

Here is another document:

Requirements of build tools:

clang: 17
cmake: 3.28
ninja: 1.11

In this document we use clang-17 installed in /usr/bin as an example.

First clone llvm and build and install the runtimes:

git clone https://github.com/llvm/llvm-project.git --depth 1
cd llvm-project
cmake -G Ninja -S runtimes -B build -DCMAKE_C_COMPILER=/usr/bin/clang-17 -DCMAKE_CXX_COMPILER=/usr/bin/clang-17 -DLIBCXX_INSTALL_MODULES=ON -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi;libunwind"
cmake --build build -- -j 12
cmake --install build --prefix <install_prefix>

Then we create an external project with root directory <project_root>. First we create <project_root>/CMakeLists.txt:

cmake_minimum_required(VERSION 3.28.0 FATAL_ERROR)

project(HelloWorld LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 23)
set(CMAKE_CXX_STANDARD_REQUIRED YES)
set(CMAKE_CXX_EXTENSIONS OFF)

include(std.cmake)

add_executable(main main.cpp)

Then the <project_root>/std.cmake:

include(FetchContent)
FetchContent_Declare(
  std_module
  URL "file://${LIBCXX_DIR}/share/libc++/v1"
  DOWNLOAD_EXTRACT_TIMESTAMP TRUE
  SYSTEM
)

if (NOT std_module_POPULATED)
  FetchContent_Populate(std_module)
endif()

add_library(std)

target_sources(std
  PUBLIC FILE_SET cxx_modules TYPE CXX_MODULES FILES
    ${std_module_SOURCE_DIR}/std.cppm
    ${std_module_SOURCE_DIR}/std.compat.cppm
)

target_include_directories(std SYSTEM PUBLIC ${LIBCXX_DIR}/include/c++/v1)

target_compile_options(std
  PRIVATE
    -Wno-reserved-module-identifier
    -Wno-reserved-user-defined-literal
)

target_compile_options(std
  PUBLIC
    -nostdinc++
)

target_link_options(std
  INTERFACE
    -nostdlib++
    -L${LIBCXX_DIR}/lib
    -Wl,-rpath,${LIBCXX_DIR}/lib
)

target_link_libraries(std
  INTERFACE
    c++
)

link_libraries(std)

Finally, we create the <project_root>/main.cpp

import std;

auto main() -> int {
    std::println("Hello, world!");
}

Please note that you need more than clang itself to build a project using modules. Specifically, you will need clang-scan-deps. For example, in Ubuntu, you need to use sudo ./llvm.sh 17 all rather than sudo ./llvm.sh 17 showed in LLVM Debian/Ubuntu nightly packages to install essential components to build this project. If you have installed all needed tools, then we can configure and build this external project:

mkdir build
cmake -S <project_root> -B build -G Ninja -DCMAKE_CXX_COMPILER=/usr/bin/clang-17 -DLIBCXX_DIR=<install_prefix>
cmake --build build
./build/main

If all goes well, now you can see:

Hello, world!

[mordante]

And I do not actually understand what you mean making a pull request. Because all my current method is here in my comment above. And there is nothing to change in the source code of llvm. All of these files are scripts and cmake file in the external project. So first, I can not make pull request of llvm's patch. And also, I think you are not satisfied with my method now. So second, I can not make a pull request of document. Finally, in general, I don't understand what kind of pull request you want me to make.

Libcxx's document of C++ std library modules in Modules in libc++ is outdated, which is compiled from Modules.rst.

I'd like a pull request for the Modules.rst, which contains the contents of the scripts you posted. That way we can provide this information to the users of libc++. In general your scripts look good, there are a few things that differ from your "typical style" I also want to test with this script. Supplying suggestions in a patch is easier. Alternatively I can create a patch based on the scripts in this bug report.

[a858438680]

And I do not actually understand what you mean making a pull request. Because all my current method is here in my comment above. And there is nothing to change in the source code of llvm. All of these files are scripts and cmake file in the external project. So first, I can not make pull request of llvm's patch. And also, I think you are not satisfied with my method now. So second, I can not make a pull request of document. Finally, in general, I don't understand what kind of pull request you want me to make.

Libcxx's document of C++ std library modules in Modules in libc++ is outdated, which is compiled from Modules.rst.

I'd like a pull request for the Modules.rst, which contains the contents of the scripts you posted. That way we can provide this information to the users of libc++. In general your scripts look good, there are a few things that differ from your "typical style" I also want to test with this script. Supplying suggestions in a patch is easier. Alternatively I can create a patch based on the scripts in this bug report.

Thanks for your clarification. Are you satisfied with my new method in general? It does not require a complete build. If you are satisfied in general, I will make a pull request as soon as possible.

[a858438680]

Hi @mordante, I have made a pull request #80601.

[ChuanqiXu9]

Given #80601 is closed, should we close this?