[libc] Find Python 3 in standalone #118871

RossComputerGuy · 2024-12-05T21:11:08Z

Python3_Executable wasn't defined when libc is trying to be built in standalone mode. Also adds a project define so CMake quits complaining.

llvmbot · 2024-12-05T21:11:44Z

@llvm/pr-subscribers-libc

Author: Tristan Ross (RossComputerGuy)

Changes

Python3_Executable wasn't defined when libc is trying to be built in standalone mode. Also adds a project define so CMake quits complaining.

Full diff: https://github.com/llvm/llvm-project/pull/118871.diff

1 Files Affected:

(modified) libc/CMakeLists.txt (+12)

diff --git a/libc/CMakeLists.txt b/libc/CMakeLists.txt
index 5196735bef4e76..cfe3e347980a37 100644
--- a/libc/CMakeLists.txt
+++ b/libc/CMakeLists.txt
@@ -8,6 +8,13 @@ endif()
 include(${LLVM_COMMON_CMAKE_UTILS}/Modules/CMakePolicy.cmake
   NO_POLICY_SCOPE)
 
+# If we are not building as a part of LLVM, build libc as an
+# standalone project, using LLVM as an external library:
+if(CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR)
+  project(libc)
+  set(LIBC_BUILT_STANDALONE TRUE)
+endif()
+
 if (LIBC_CMAKE_VERBOSE_LOGGING)
   get_directory_property(LIBC_OLD_PREPROCESSOR_DEFS COMPILE_DEFINITIONS)
   foreach(OLD_DEF ${LIBC_OLD_PREPROCESSOR_DEFS})
@@ -30,6 +37,11 @@ endif()
 # Default to C++17
 set(CMAKE_CXX_STANDARD 17)
 
+if(LIBC_BUILT_STANDALONE)
+  find_package(Python3 ${LLVM_MINIMUM_PYTHON_VERSION} REQUIRED
+    COMPONENTS Interpreter)
+endif()
+
 list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/modules")
 
 # The top-level source directory.

jhuber6

Isn't there some external use of libc more standalone? I'll defer to @nickdesaulniers.

nickdesaulniers · 2024-12-05T21:16:05Z

Thanks for the patch! How are you invoking the build @RossComputerGuy ? (i.e. what explicit command line commands are you running)?

RossComputerGuy · 2024-12-05T21:18:13Z

I'm running with these flags:

-GNinja -DCMAKE_FIND_USE_SYSTEM_PACKAGE_REGISTRY=OFF -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTING=OFF -DCMAKE_INSTALL_LOCALEDIR=/nix/store/lfasxjrpvnnwd8imf3wfwf784mkirc3w-libc-20.0.0-unstable-2024-12-05/share/locale -DCMAKE_INSTALL_LIBEXECDIR=/nix/store/lfasxjrpvnnwd8imf3wfwf784mkirc3w-libc-20.0.0-unstable-2024-12-05/libexec -DCMAKE_INSTALL_LIBDIR=/nix/store/lfasxjrpvnnwd8imf3wfwf784mkirc3w-libc-20.0.0-unstable-2024-12-05/lib -DCMAKE_INSTALL_DOCDIR=/nix/store/lfasxjrpvnnwd8imf3wfwf784mkirc3w-libc-20.0.0-unstable-2024-12-05/share/doc/libc -DCMAKE_INSTALL_INFODIR=/nix/store/lfasxjrpvnnwd8imf3wfwf784mkirc3w-libc-20.0.0-unstable-2024-12-05/share/info -DCMAKE_INSTALL_MANDIR=/nix/store/lfasxjrpvnnwd8imf3wfwf784mkirc3w-libc-20.0.0-unstable-2024-12-05/share/man -DCMAKE_INSTALL_OLDINCLUDEDIR=/nix/store/lfasxjrpvnnwd8imf3wfwf784mkirc3w-libc-20.0.0-unstable-2024-12-05/include -DCMAKE_INSTALL_INCLUDEDIR=/nix/store/lfasxjrpvnnwd8imf3wfwf784mkirc3w-libc-20.0.0-unstable-2024-12-05/include -DCMAKE_INSTALL_SBINDIR=/nix/store/lfasxjrpvnnwd8imf3wfwf784mkirc3w-libc-20.0.0-unstable-2024-12-05/sbin -DCMAKE_INSTALL_BINDIR=/nix/store/lfasxjrpvnnwd8imf3wfwf784mkirc3w-libc-20.0.0-unstable-2024-12-05/bin -DCMAKE_INSTALL_NAME_DIR=/nix/store/lfasxjrpvnnwd8imf3wfwf784mkirc3w-libc-20.0.0-unstable-2024-12-05/lib -DCMAKE_POLICY_DEFAULT_CMP0025=NEW -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_STRIP=/nix/store/2r2xi5pbg29bsmqywsm5zgl8l7adky4i-gcc-wrapper-13.3.0/bin/strip -DCMAKE_RANLIB=/nix/store/2r2xi5pbg29bsmqywsm5zgl8l7adky4i-gcc-wrapper-13.3.0/bin/ranlib -DCMAKE_AR=/nix/store/2r2xi5pbg29bsmqywsm5zgl8l7adky4i-gcc-wrapper-13.3.0/bin/ar -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_INSTALL_PREFIX=/nix/store/lfasxjrpvnnwd8imf3wfwf784mkirc3w-libc-20.0.0-unstable-2024-12-05 -DLLVM_LIBC_FULL_BUILD:BOOL=TRUE -DPython3_EXECUTABLE:STRING=/nix/store/imz4prdh762l7jvwak2s7pqpq6hy5fyy-python3-3.12.7/bin/python3.12

I have to specify Python3_EXECUTABLE since it tries executing the yaml_to_classes.py script in a way that makes Nix not happy.

nickdesaulniers · 2024-12-05T21:25:22Z

What's the full cmake command? Specifically, what directory are you using as the root for cmake?

RossComputerGuy · 2024-12-05T21:30:54Z

I'm not sure how to extract the full on CMake command but I know it runs with cmake ... That command is ran inside libc/build which that directory is created.

michaelrj-google · 2024-12-05T21:47:17Z

libc/CMakeLists.txt

+if(LIBC_BUILT_STANDALONE)
+  find_package(Python3 ${LLVM_MINIMUM_PYTHON_VERSION} REQUIRED
+    COMPONENTS Interpreter)
+endif()


we also need pyyaml for newhdrgen. Not sure if that needs to be specified in the cmake, but adding a comment at least would be useful.

I'll add a comment about it at least, I'm not sure how we could add a check inside CMake to ensure the module exists.

nickdesaulniers · 2024-12-05T22:27:02Z

Right, so in #78479, I'm trying to move to only documenting+supporting using llvm-project/runtimes dir as the root for cmake for building llvm-libc.

It sounds like you (and @SchrodingerZhu ) are trying to use llvm-project/libc/ as the root for cmake for building llvm-libc.

Is there a reason why you cannot use llvm-project/runtimes/ as the root for cmake for llvm-libc?

i.e.

$ cd llvm-project
$ mkdir build
$ cd build
$ cmake ../runtimes ...
<rather than>
$ cmake ../libc ...

RossComputerGuy · 2024-12-05T22:30:14Z

Oh, I wasn't aware of that change. I'm not sure what would be stopping Nixpkgs from using that once it's merged. Would this allow for building completely separate from the rest of things just like what is possible with llvm, clang, and the various other projects inside of LLVM?

nickdesaulniers · 2024-12-05T22:33:59Z

Would this allow for building completely separate from the rest of things just like what is possible with llvm, clang, and the various other projects inside of LLVM?

Yes. We recently removed a dependency on tablegen, which implied a depedency on llvm. This should let us bootstrap clang itself from the libc (avoiding having to build llvm twice, one for libc, again for clang).

SchrodingerZhu · 2024-12-05T23:56:47Z

In my case, it is just that llvm-project is too large to be packaged in a single rust package. But that experiment did not go out well.

SchrodingerZhu · 2024-12-05T23:59:17Z

See https://github.com/SchrodingerZhu/rust-linux-llvm/blob/main/llvmlibc-src/src/update.sh for how to build libc fully out of tree. It may be broken already as @jhuber6 has reverted some needed changes.

nickdesaulniers · 2024-12-09T18:27:44Z

But that experiment did not go out well.

In that case, if there are no existing use cases, I'd prefer NOT to add more ways to build llvm-libc. Our documentation is already too confusing; we're actively trying to simplify it. We're also not continuously testing such a method of building, so we could regress it by accident.

@RossComputerGuy can you please confirm whether my suggestion above (using runtimes dir, rather than libc dir) works for you?

RossComputerGuy · 2024-12-10T04:18:15Z

@nickdesaulniers I did try it but I've ran into problems:

libc> CMake Error at CMakeLists.txt:174 (include):
libc>   include could not find requested file:
libc>
libc>     AddLLVM
libc>
libc>
libc> CMake Error at CMakeLists.txt:175 (include):
libc>   include could not find requested file:
libc>
libc>     HandleLLVMOptions
libc>
libc>
libc> -- Found Python3: /nix/store/imz4prdh762l7jvwak2s7pqpq6hy5fyy-python3-3.12.7/bin/python3.12 (found version "3.12.7") found components: Interpreter
libc> CMake Error at CMakeLists.txt:183 (include):
libc>   include could not find requested file:
libc>
libc>     GetHostTriple
libc>
libc>
libc> CMake Error at CMakeLists.txt:184 (get_host_triple):
libc>   Unknown CMake command "get_host_triple".

SchrodingerZhu · 2024-12-10T04:22:07Z

It can be fixed by adding in the cmake modules under llvm's root and llvm/cmake.

RossComputerGuy · 2024-12-10T04:25:40Z

Yeah, that did make it work.

carlocab · 2024-12-10T04:53:28Z

libc/CMakeLists.txt

+if(CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR)
+  project(libc)
+  set(LIBC_BUILT_STANDALONE TRUE)
+endif()


Given that libc doesn't support builds rooted in the libc directory (and I don't think we want to add support for it), maybe we should be erroring out here instead.

I don't think so, would we stop supporting standalone builds of everything else (clang, llvm, bolt, lldb, etc.) unless they use runtimes? I don't really think an error is necessary.

No, but that's because clang, lldb, and maybe bolt, etc do support standalone builds rooted in their subdirectory. libc does not.

So erroring out here has no implications for those other subprojects, nor does the fact that they support standalone builds imply anything for what libc should do about them.

A better comparison would be libcxx, which I believe does (or, at least, should) error out on builds rooted in the libcxx directory.

I think a warning (or error) would be a very user-friendly feature to add. Then we can point users in the right direction if they make a mistake.

Opened #119426 for this.

Building with the source directory rooted in the libc subdirectory isn't tested in CI and can lead to subtle build problems (cf. llvm#118871). Let's fail fast with a helpful error message instead to help users configure libc correctly.

Building with the source directory rooted in the libc subdirectory isn't tested in CI and can lead to subtle build problems (cf. llvm#118871). Let's fail fast with a helpful error message instead to help users configure libc correctly. Co-authored-by: Nick Desaulniers <[email protected]>

nickdesaulniers · 2024-12-10T20:48:32Z

Closing in favor of #119426.

Building with the source directory rooted in the libc subdirectory isn't tested in CI and can lead to subtle build problems (cf. #118871). Let's fail fast with a helpful error message instead to help users configure libc correctly. Co-authored-by: Nick Desaulniers <[email protected]>

llvmbot added the libc label Dec 5, 2024

jhuber6 reviewed Dec 5, 2024

View reviewed changes

nickdesaulniers requested a review from SchrodingerZhu December 5, 2024 21:15

michaelrj-google reviewed Dec 5, 2024

View reviewed changes

[libc] Find Python 3 in standalone

268a37e

RossComputerGuy force-pushed the feat/llvm-libc-fixes branch from 275a03c to 268a37e Compare December 5, 2024 21:58

carlocab reviewed Dec 10, 2024

View reviewed changes

carlocab mentioned this pull request Dec 10, 2024

[libc] Fail fast when building standalone #119426

Merged

nickdesaulniers closed this Dec 10, 2024

RossComputerGuy deleted the feat/llvm-libc-fixes branch December 10, 2024 21:25

[libc] Find Python 3 in standalone #118871

[libc] Find Python 3 in standalone #118871

Uh oh!

Conversation

RossComputerGuy commented Dec 5, 2024

Uh oh!

llvmbot commented Dec 5, 2024

Uh oh!

jhuber6 left a comment

Choose a reason for hiding this comment

Uh oh!

nickdesaulniers commented Dec 5, 2024

Uh oh!

RossComputerGuy commented Dec 5, 2024

Uh oh!

nickdesaulniers commented Dec 5, 2024

Uh oh!

RossComputerGuy commented Dec 5, 2024

Uh oh!

michaelrj-google Dec 5, 2024

Choose a reason for hiding this comment

Uh oh!

RossComputerGuy Dec 5, 2024

Choose a reason for hiding this comment

Uh oh!

nickdesaulniers commented Dec 5, 2024

Uh oh!

RossComputerGuy commented Dec 5, 2024

Uh oh!

nickdesaulniers commented Dec 5, 2024

Uh oh!

SchrodingerZhu commented Dec 5, 2024

Uh oh!

SchrodingerZhu commented Dec 5, 2024

Uh oh!

nickdesaulniers commented Dec 9, 2024

Uh oh!

RossComputerGuy commented Dec 10, 2024

Uh oh!

SchrodingerZhu commented Dec 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RossComputerGuy commented Dec 10, 2024

Uh oh!

carlocab Dec 10, 2024

Choose a reason for hiding this comment

Uh oh!

RossComputerGuy Dec 10, 2024

Choose a reason for hiding this comment

Uh oh!

carlocab Dec 10, 2024

Choose a reason for hiding this comment

Uh oh!

nickdesaulniers Dec 10, 2024

Choose a reason for hiding this comment

Uh oh!

carlocab Dec 10, 2024

Choose a reason for hiding this comment

Uh oh!

nickdesaulniers commented Dec 10, 2024

Uh oh!

Uh oh!

SchrodingerZhu commented Dec 10, 2024 •

edited

Loading