Skip to content

Reland Print library module manifest path again #84881

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 17, 2024

Conversation

ChuanqiXu9
Copy link
Member

Following of #82160

The reason why the above PR fails is that the --sysroot has lower priority than the libc++ built from the same source. On the one hand, it matches the codes behavior. We will add the built libc++ project paths in the ToolChain class. But we will only add the path related to sysroot in Linux class, which is derived from the ToolChain classes. So the paths of just built libc++ is in the front of the paths relative to sysroot. On the other hand, the behavior should be good from the higher level. Since the just built libc++ has the same version number with the just built clang, so it makes sense that these 2 compilers just matches.

So for patch it self, I hacked it by using resource dir in the test since the resource dir has the higher priority, which is not strongly correct since we won't do that in practice.

@kaz7 would you like to test on your environment to avoid this get reverted again?

On the libc++ side, it shows that it lacks a modules.json file for the just built libc++ directory. If we don't have that, it will be problematic to use std modules from the just built clang and libc++ pair. Then it is not good. And I feel it may be problematic for future compiler/standard library developers. So I feel this is somewhat a libc++ issue that need to be fixed.

Also if we don't like the hacked test in the current patch, we must wait for libc++ to fix this to proceed. But I feel this is somewhat odd since the test of clang shouldn't dependent on libc++.

CC: @mordante

mordante and others added 2 commits March 12, 2024 10:49
…2160)

This implements a way for the compiler to find the modules.json
associated with the C++23 Standard library modules.

This is based on a discussion in SG15. At the moment no Standard library
installs this manifest. llvm#75741 adds this feature in libc++.

This reverts commit 82f424f.

Disables the tests on non-X86 platforms as suggested.
@ChuanqiXu9 ChuanqiXu9 added the clang:modules C++20 modules and Clang Header Modules label Mar 12, 2024
@ChuanqiXu9 ChuanqiXu9 requested review from mordante and kaz7 March 12, 2024 07:29
@llvmbot llvmbot added clang Clang issues not falling into any other category clang:driver 'clang' and 'clang++' user-facing binaries. Not 'clang-cl' labels Mar 12, 2024
@llvmbot
Copy link
Member

llvmbot commented Mar 12, 2024

@llvm/pr-subscribers-clang
@llvm/pr-subscribers-clang-driver

@llvm/pr-subscribers-clang-modules

Author: Chuanqi Xu (ChuanqiXu9)

Changes

Following of #82160

The reason why the above PR fails is that the --sysroot has lower priority than the libc++ built from the same source. On the one hand, it matches the codes behavior. We will add the built libc++ project paths in the ToolChain class. But we will only add the path related to sysroot in Linux class, which is derived from the ToolChain classes. So the paths of just built libc++ is in the front of the paths relative to sysroot. On the other hand, the behavior should be good from the higher level. Since the just built libc++ has the same version number with the just built clang, so it makes sense that these 2 compilers just matches.

So for patch it self, I hacked it by using resource dir in the test since the resource dir has the higher priority, which is not strongly correct since we won't do that in practice.

@kaz7 would you like to test on your environment to avoid this get reverted again?

On the libc++ side, it shows that it lacks a modules.json file for the just built libc++ directory. If we don't have that, it will be problematic to use std modules from the just built clang and libc++ pair. Then it is not good. And I feel it may be problematic for future compiler/standard library developers. So I feel this is somewhat a libc++ issue that need to be fixed.

Also if we don't like the hacked test in the current patch, we must wait for libc++ to fix this to proceed. But I feel this is somewhat odd since the test of clang shouldn't dependent on libc++.

CC: @mordante


Full diff: https://github.com/llvm/llvm-project/pull/84881.diff

4 Files Affected:

  • (modified) clang/include/clang/Driver/Driver.h (+10)
  • (modified) clang/include/clang/Driver/Options.td (+3)
  • (modified) clang/lib/Driver/Driver.cpp (+44)
  • (added) clang/test/Driver/modules-print-library-module-manifest-path.cpp (+36)
diff --git a/clang/include/clang/Driver/Driver.h b/clang/include/clang/Driver/Driver.h
index c4cab360bab3bb..bcf8c1295f2ddf 100644
--- a/clang/include/clang/Driver/Driver.h
+++ b/clang/include/clang/Driver/Driver.h
@@ -615,6 +615,16 @@ class Driver {
   // FIXME: This should be in CompilationInfo.
   std::string GetProgramPath(StringRef Name, const ToolChain &TC) const;
 
+  /// Lookup the path to the Standard library module manifest.
+  ///
+  /// \param C - The compilation.
+  /// \param TC - The tool chain for additional information on
+  /// directories to search.
+  //
+  // FIXME: This should be in CompilationInfo.
+  std::string GetStdModuleManifestPath(const Compilation &C,
+                                       const ToolChain &TC) const;
+
   /// HandleAutocompletions - Handle --autocomplete by searching and printing
   /// possible flags, descriptions, and its arguments.
   void HandleAutocompletions(StringRef PassedFlags) const;
diff --git a/clang/include/clang/Driver/Options.td b/clang/include/clang/Driver/Options.td
index aca8c9b0d5487a..6e7f142789c994 100644
--- a/clang/include/clang/Driver/Options.td
+++ b/clang/include/clang/Driver/Options.td
@@ -5374,6 +5374,9 @@ def print_resource_dir : Flag<["-", "--"], "print-resource-dir">,
 def print_search_dirs : Flag<["-", "--"], "print-search-dirs">,
   HelpText<"Print the paths used for finding libraries and programs">,
   Visibility<[ClangOption, CLOption]>;
+def print_std_module_manifest_path : Flag<["-", "--"], "print-library-module-manifest-path">,
+  HelpText<"Print the path for the C++ Standard library module manifest">,
+  Visibility<[ClangOption, CLOption]>;
 def print_targets : Flag<["-", "--"], "print-targets">,
   HelpText<"Print the registered targets">,
   Visibility<[ClangOption, CLOption]>;
diff --git a/clang/lib/Driver/Driver.cpp b/clang/lib/Driver/Driver.cpp
index 190782a79a2456..eae27769b73ec0 100644
--- a/clang/lib/Driver/Driver.cpp
+++ b/clang/lib/Driver/Driver.cpp
@@ -2196,6 +2196,12 @@ bool Driver::HandleImmediateArgs(const Compilation &C) {
     return false;
   }
 
+  if (C.getArgs().hasArg(options::OPT_print_std_module_manifest_path)) {
+    llvm::outs() << GetStdModuleManifestPath(C, C.getDefaultToolChain())
+                 << '\n';
+    return false;
+  }
+
   if (C.getArgs().hasArg(options::OPT_print_runtime_dir)) {
     if (std::optional<std::string> RuntimePath = TC.getRuntimePath())
       llvm::outs() << *RuntimePath << '\n';
@@ -6185,6 +6191,44 @@ std::string Driver::GetProgramPath(StringRef Name, const ToolChain &TC) const {
   return std::string(Name);
 }
 
+std::string Driver::GetStdModuleManifestPath(const Compilation &C,
+                                             const ToolChain &TC) const {
+  std::string error = "<NOT PRESENT>";
+
+  switch (TC.GetCXXStdlibType(C.getArgs())) {
+  case ToolChain::CST_Libcxx: {
+    std::string lib = GetFilePath("libc++.so", TC);
+
+    // Note when there are multiple flavours of libc++ the module json needs to
+    // look at the command-line arguments for the proper json.
+    // These flavours do not exist at the moment, but there are plans to
+    // provide a variant that is built with sanitizer instrumentation enabled.
+
+    // For example
+    //  StringRef modules = [&] {
+    //    const SanitizerArgs &Sanitize = TC.getSanitizerArgs(C.getArgs());
+    //    if (Sanitize.needsAsanRt())
+    //      return "modules-asan.json";
+    //    return "modules.json";
+    //  }();
+
+    SmallString<128> path(lib.begin(), lib.end());
+    llvm::sys::path::remove_filename(path);
+    llvm::sys::path::append(path, "modules.json");
+    if (TC.getVFS().exists(path))
+      return static_cast<std::string>(path);
+
+    return error;
+  }
+
+  case ToolChain::CST_Libstdcxx:
+    // libstdc++ does not provide Standard library modules yet.
+    return error;
+  }
+
+  return error;
+}
+
 std::string Driver::GetTemporaryPath(StringRef Prefix, StringRef Suffix) const {
   SmallString<128> Path;
   std::error_code EC = llvm::sys::fs::createTemporaryFile(Prefix, Suffix, Path);
diff --git a/clang/test/Driver/modules-print-library-module-manifest-path.cpp b/clang/test/Driver/modules-print-library-module-manifest-path.cpp
new file mode 100644
index 00000000000000..24797002b80f53
--- /dev/null
+++ b/clang/test/Driver/modules-print-library-module-manifest-path.cpp
@@ -0,0 +1,36 @@
+// Test that -print-library-module-manifest-path finds the correct file.
+
+// RUN: rm -rf %t && split-file %s %t && cd %t
+// RUN: mkdir -p %t/Inputs/usr/lib/x86_64-linux-gnu
+// RUN: touch %t/Inputs/usr/lib/x86_64-linux-gnu/libc++.so
+
+// RUN: %clang -print-library-module-manifest-path \
+// RUN:     -stdlib=libc++ \
+// RUN:     -resource-dir=%t/Inputs/usr/lib/x86_64-linux-gnu \
+// RUN:     --target=x86_64-linux-gnu 2>&1 \
+// RUN:   | FileCheck libcxx-no-module-json.cpp
+
+// RUN: touch %t/Inputs/usr/lib/x86_64-linux-gnu/modules.json
+// RUN: %clang -print-library-module-manifest-path \
+// RUN:     -stdlib=libc++ \
+// RUN:     -resource-dir=%t/Inputs/usr/lib/x86_64-linux-gnu \
+// RUN:     --target=x86_64-linux-gnu 2>&1 \
+// RUN:   | FileCheck libcxx.cpp
+
+// RUN: %clang -print-library-module-manifest-path \
+// RUN:     -stdlib=libstdc++ \
+// RUN:     -resource-dir=%t/Inputs/usr/lib/x86_64-linux-gnu \
+// RUN:     --target=x86_64-linux-gnu 2>&1 \
+// RUN:   | FileCheck libstdcxx.cpp
+
+//--- libcxx-no-module-json.cpp
+
+// CHECK: <NOT PRESENT>
+
+//--- libcxx.cpp
+
+// CHECK: {{.*}}/Inputs/usr/lib/x86_64-linux-gnu{{/|\\}}modules.json
+
+//--- libstdcxx.cpp
+
+// CHECK: <NOT PRESENT>

Copy link
Contributor

@kaz7 kaz7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for updating test scripts. It works fine in the situation I explained before. It works fine under VE buildbot configuration too.

Copy link
Member

@mordante mordante left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking it up @ChuanqiXu9 and thanks for testing @kaz7.

@ChuanqiXu9 ChuanqiXu9 merged commit 0e1e1fc into llvm:main Mar 17, 2024
@ChuanqiXu9
Copy link
Member Author

I'll backport this tomorrow if no revert request shows up.

@Arthapz
Copy link

Arthapz commented Mar 17, 2024

How do we use this ?
i tried

> /opt/llvm-git/usr/bin/clang -std=c++23 -stdlib=libc++ -print-library-module-manifest-path
<NOT PRESENT>

and

> /opt/llvm-git/usr/bin/clang -std=c++23 -stdlib=libc++ -print-library-module-manifest-path -resource-dir=/opt/llvm-git/usr/lib
<NOT PRESENT>

but no manifest path is returned

> fd .json /opt/llvm-git/usr/lib
/opt/llvm-git/usr/lib/libc++.modules.json

@mordante
Copy link
Member

The file we're looking for is modules.json; Renaming it libc++.modules.json like .so / .a file might be a better idea which could avoid name clashes when installed in /usr/lib.

@ChuanqiXu9
Copy link
Member Author

The file we're looking for is modules.json; Renaming it libc++.modules.json like .so / .a file might be a better idea which could avoid name clashes when installed in /usr/lib.

BTW, how do you feel about the paragraph. I thought it as a defect in some level.

@ChuanqiXu9 ChuanqiXu9 added this to the LLVM 18.X Release milestone Mar 18, 2024
@ChuanqiXu9
Copy link
Member Author

/cherry-pick 0e1e1fc

@ChuanqiXu9 ChuanqiXu9 deleted the PrintLibraryModuleManifestPath branch March 18, 2024 12:29
@Arthapz
Copy link

Arthapz commented Mar 18, 2024

The file we're looking for is modules.json; Renaming it libc++.modules.json like .so / .a file might be a better idea which could avoid name clashes when installed in /usr/lib.

but i didn't rename it, it was with the libc++ prefix directly :/

llvmbot pushed a commit to llvmbot/llvm-project that referenced this pull request Mar 18, 2024
Following of llvm#82160

The reason why the above PR fails is that the `--sysroot` has lower
priority than the libc++ built from the same source. On the one hand, it
matches the codes behavior. We will add the built libc++ project paths
in the ToolChain class. But we will only add the path related to sysroot
in Linux class, which is derived from the ToolChain classes. So the
paths of just built libc++ is in the front of the paths relative to
sysroot. On the other hand, the behavior should be good from the higher
level. Since the just built libc++ has the same version number with the
just built clang, so it makes sense that these 2 compilers just matches.

So for patch it self, I hacked it by using resource dir in the test
since the resource dir has the higher priority, which is not strongly
correct since we won't do that in practice.

@kaz7 would you like to test on your environment to avoid this get
reverted again?

On the libc++ side, it shows that it lacks a `modules.json` file for the
just built libc++ directory. If we don't have that, it will be
problematic to use std modules from the just built clang and libc++
pair. Then it is not good. And I feel it may be problematic for future
compiler/standard library developers. So I feel this is somewhat a
libc++ issue that need to be fixed.

Also if we don't like the hacked test in the current patch, we must wait
for libc++ to fix this to proceed. But I feel this is somewhat odd since
the test of clang shouldn't dependent on libc++.

CC: @mordante

---------

Co-authored-by: Mark de Wever <[email protected]>
(cherry picked from commit 0e1e1fc)
@llvmbot
Copy link
Member

llvmbot commented Mar 18, 2024

/pull-request #85637

@mordante
Copy link
Member

The file we're looking for is modules.json; Renaming it libc++.modules.json like .so / .a file might be a better idea which could avoid name clashes when installed in /usr/lib.

but i didn't rename it, it was with the libc++ prefix directly :/

Good point I did :-/ It seems we originally talked about modules.json and later mentioned using the libname. @ChuanqiXu9 I prefer to keep the current name libc++.modules.json and adjust clang. WDYT?

@ChuanqiXu9
Copy link
Member Author

The file we're looking for is modules.json; Renaming it libc++.modules.json like .so / .a file might be a better idea which could avoid name clashes when installed in /usr/lib.

but i didn't rename it, it was with the libc++ prefix directly :/

Good point I did :-/ It seems we originally talked about modules.json and later mentioned using the libname. @ChuanqiXu9 I prefer to keep the current name libc++.modules.json and adjust clang. WDYT?

Could you elaborate the reason more? I feel like renaming libc++.modules.json to modules.json is straightforward. Maybe I didn't take part it in the old disscussion or I missed : )

@mordante
Copy link
Member

The file we're looking for is modules.json; Renaming it libc++.modules.json like .so / .a file might be a better idea which could avoid name clashes when installed in /usr/lib.

but i didn't rename it, it was with the libc++ prefix directly :/

Good point I did :-/ It seems we originally talked about modules.json and later mentioned using the libname. @ChuanqiXu9 I prefer to keep the current name libc++.modules.json and adjust clang. WDYT?

Could you elaborate the reason more? I feel like renaming libc++.modules.json to modules.json is straightforward. Maybe I didn't take part it in the old disscussion or I missed : )

It would allow to install libstdc++ and libc++ in the same lib directory without "fighting" who owns modules.json. Also if we want to provide libc++-asan.so in the future we can provide libc++-asan.modules.json. Whether we need a different module.json for ASan is unknown at the moment, but I can imagine we have additional compiler or linker flags for ASan.

@ChuanqiXu9
Copy link
Member Author

ChuanqiXu9 commented Mar 19, 2024

The file we're looking for is modules.json; Renaming it libc++.modules.json like .so / .a file might be a better idea which could avoid name clashes when installed in /usr/lib.

but i didn't rename it, it was with the libc++ prefix directly :/

Good point I did :-/ It seems we originally talked about modules.json and later mentioned using the libname. @ChuanqiXu9 I prefer to keep the current name libc++.modules.json and adjust clang. WDYT?

Could you elaborate the reason more? I feel like renaming libc++.modules.json to modules.json is straightforward. Maybe I didn't take part it in the old disscussion or I missed : )

It would allow to install libstdc++ and libc++ in the same lib directory without "fighting" who owns modules.json. Also if we want to provide libc++-asan.so in the future we can provide libc++-asan.modules.json. Whether we need a different module.json for ASan is unknown at the moment, but I can imagine we have additional compiler or linker flags for ASan.

hmmmm, I can accept that. Then the patch itself is not incorrect... Since it looks like we'd never look for modules.json actually, right? Let's adjust the implementation in clang. Also possibly, we can't/shouldn't backport this 18. How do you think?

Would you like to pick up this? I feel you may be more appropriate since you understand the environment better. I'm also like to take it if you don't have the time.

@mordante
Copy link
Member

Build system don't use this yet (since it wasn't available). I know CMake is working on implementing modules using libc++. Looking at the stage of the release I feel less comfortable to propose new features. So I think the feature should not be back ported.

I'm happy to pick this up, I expect to have time for it tomorrow.

@stellar-aria
Copy link

Can we also have a fallback check for stdlibs that don't have a .so version (such as embedded)? Something as simple as:

    std::string lib = GetFilePath("libc++.so", TC);
    if (lib.empty())
      lib = GetFilePath("libc++.a", TC);

@mordante
Copy link
Member

Can we also have a fallback check for stdlibs that don't have a .so version (such as embedded)? Something as simple as:

    std::string lib = GetFilePath("libc++.so", TC);
    if (lib.empty())
      lib = GetFilePath("libc++.a", TC);

Excellent point! thanks!

llvmbot pushed a commit to llvmbot/llvm-project that referenced this pull request Mar 19, 2024
Following of llvm#82160

The reason why the above PR fails is that the `--sysroot` has lower
priority than the libc++ built from the same source. On the one hand, it
matches the codes behavior. We will add the built libc++ project paths
in the ToolChain class. But we will only add the path related to sysroot
in Linux class, which is derived from the ToolChain classes. So the
paths of just built libc++ is in the front of the paths relative to
sysroot. On the other hand, the behavior should be good from the higher
level. Since the just built libc++ has the same version number with the
just built clang, so it makes sense that these 2 compilers just matches.

So for patch it self, I hacked it by using resource dir in the test
since the resource dir has the higher priority, which is not strongly
correct since we won't do that in practice.

@kaz7 would you like to test on your environment to avoid this get
reverted again?

On the libc++ side, it shows that it lacks a `modules.json` file for the
just built libc++ directory. If we don't have that, it will be
problematic to use std modules from the just built clang and libc++
pair. Then it is not good. And I feel it may be problematic for future
compiler/standard library developers. So I feel this is somewhat a
libc++ issue that need to be fixed.

Also if we don't like the hacked test in the current patch, we must wait
for libc++ to fix this to proceed. But I feel this is somewhat odd since
the test of clang shouldn't dependent on libc++.

CC: @mordante

---------

Co-authored-by: Mark de Wever <[email protected]>
(cherry picked from commit 0e1e1fc)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
clang:driver 'clang' and 'clang++' user-facing binaries. Not 'clang-cl' clang:modules C++20 modules and Clang Header Modules clang Clang issues not falling into any other category
Projects
Development

Successfully merging this pull request may close these issues.

6 participants