[lldb][test] Fix TestStdCXXDisassembly test

[slydiman]

The patch #98694 was not enough. This test is still failed on the buildbot https://lab.llvm.org/staging/#/builders/195/builds/4438
Use USE_LIBSTDCPP := 1 instead for non Darwin OS and skip the test if libstdc++.so is missing.

[llvmbot]

@llvm/pr-subscribers-lldb

Author: Dmitry Vasilyev (slydiman)

Changes

The patch #98694 was not enough. This test is still failed on the buildbot https://lab.llvm.org/staging/#/builders/195/builds/4438 Use USE_LIBSTDCPP := 1 instead.

---

Full diff: https://github.com/llvm/llvm-project/pull/112357.diff

1 Files Affected:

• (modified) lldb/test/API/lang/cpp/stl/Makefile (+1-1)

diff --git a/lldb/test/API/lang/cpp/stl/Makefile b/lldb/test/API/lang/cpp/stl/Makefile
index 8534fa9b00209e..bf8e6b8703f368 100644
--- a/lldb/test/API/lang/cpp/stl/Makefile
+++ b/lldb/test/API/lang/cpp/stl/Makefile
@@ -1,5 +1,5 @@
 CXX_SOURCES := main.cpp
 
-USE_SYSTEM_STDLIB := 1
+USE_LIBSTDCPP := 1
 
 include Makefile.rules

[Michael137]

So the problem is that because we statically link we fail to find a libstdc++.so? Won't setting USE_LIBSTDCPP be an issue on Darwin?

Generally, why is this test trying to do disassembly on every STL function? Can't really tell if this was actually STL specific from the git history. I guess as a stress-test it seems fine, but maybe we could just skip the test if we can't locate the module? Instead of messing around with the Makefile variables to try and locate the STL.

[slydiman]

maybe we could just skip the test if we can't locate the module?

Can you suggest how to do that (locate the module)?

Won't setting USE_LIBSTDCPP be an issue on Darwin?

I can't test on Darwin. Maybe the following?

ifneq ($(OS),Darwin)
    USE_LIBSTDCPP := 1
else
    USE_SYSTEM_STDLIB := 1
endif

[Michael137]

maybe we could just skip the test if we can't locate the module?

Can you suggest how to do that (locate the module)?

In TestStdCXXDisassembly.py, instead of,

self.expect(                                                            
    lib_stdcxx, "Libraray StdC++ is located", exe=False, substrs=["lib"]
)                                                                       

Do,

if "lib" not in libstd_cxx:
  self.skipTest("This test requires libstdc++.so or libc++.dylib in the target's module list.")

Won't setting USE_LIBSTDCPP be an issue on Darwin?
I can't test on Darwin. Maybe the following?

ifneq ($(OS),Darwin)
    USE_LIBSTDCPP := 1
else
    USE_SYSTEM_STDLIB := 1
endif

Yea that would be an alternative. Don't really have a preference tbh, though the former seems more robust. @labath @DavidSpickett any thoughts?

[slydiman]

@Michael137
Thanks. I have updated the patch with both solutions to be sure.

[Michael137]

Doing both seems fine, thanks!

Can we change the Darwin case to USE_LIBCPP though? The less reliance on the system stdlib the better i think.

Also lets check that the PR CI actually does run the test (aka doesn't trigger the skip added here)

[slydiman]

Can we change the Darwin case to USE_LIBCPP though? The less reliance on the system stdlib the better i think.

I cannot test it on Darwin, so I would leave that to the Darwin experts.

Also lets check that the PR CI actually does run the test (aka doesn't trigger the skip added here)

The test TestStdCXXDisassembly seems passed in PR CI. I'm going to merge this PR.

[zeroomega]

We are seeing test failures on lldb-api :: lang/cpp/stl/TestSTL.py after this patch is landed.

Error when building test subject.

Build Command:
/b/s/w/ir/x/w/cipd/bin/make VPATH=/b/s/w/ir/x/w/llvm-llvm-project/lldb/test/API/lang/cpp/stl -C /b/s/w/ir/x/w/llvm_build/lldb-test-build.noindex/lang/cpp/stl/TestSTL.test_SBType_template_aspects_dwarf -I /b/s/w/ir/x/w/llvm-llvm-project/lldb/test/API/lang/cpp/stl -I /b/s/w/ir/x/w/llvm-llvm-project/lldb/packages/Python/lldbsuite/test/make -f /b/s/w/ir/x/w/llvm-llvm-project/lldb/test/API/lang/cpp/stl/Makefile MAKE_DSYM=NO all ARCH=x86_64 CC=/b/s/w/ir/x/w/cipd/clang/bin/clang CC_TYPE=clang CXX=/b/s/w/ir/x/w/cipd/clang/bin/clang++ LLVM_AR=/b/s/w/ir/x/w/llvm_build/./bin/llvm-ar AR=/b/s/w/ir/x/w/llvm_build/./bin/llvm-ar OBJCOPY=/b/s/w/ir/x/w/llvm_build/./bin/llvm-objcopy STRIP=/b/s/w/ir/x/w/llvm_build/./bin/llvm-strip ARCHIVER=/b/s/w/ir/x/w/llvm_build/./bin/llvm-ar DWP=/b/s/w/ir/x/w/llvm_build/./bin/llvm-dwp CLANG_MODULE_CACHE_DIR=/b/s/w/ir/x/w/llvm_build/lldb-test-build.noindex/module-cache-clang/lldb-api LLDB_OBJ_ROOT=/b/s/w/ir/x/w/llvm_build/tools/lldb OS=Linux HOST_OS=Linux

Build Command Output:
make: Entering directory '/b/s/w/ir/x/w/llvm_build/lldb-test-build.noindex/lang/cpp/stl/TestSTL.test_SBType_template_aspects_dwarf'
/b/s/w/ir/x/w/cipd/clang/bin/clang++  -std=c++11 -g -O0  -m64 -I/b/s/w/ir/x/w/llvm-llvm-project/lldb/packages/Python/lldbsuite/test/make/../../../../..//include -I/b/s/w/ir/x/w/llvm_build/tools/lldb/include -I/b/s/w/ir/x/w/llvm-llvm-project/lldb/test/API/lang/cpp/stl -I/b/s/w/ir/x/w/llvm-llvm-project/lldb/packages/Python/lldbsuite/test/make -include /b/s/w/ir/x/w/llvm-llvm-project/lldb/packages/Python/lldbsuite/test/make/test_common.h -fno-limit-debug-info    -stdlib=libstdc++   --driver-mode=g++ -MT main.o -MD -MP -MF main.d -c -o main.o /b/s/w/ir/x/w/llvm-llvm-project/lldb/test/API/lang/cpp/stl/main.cpp
/b/s/w/ir/x/w/llvm-llvm-project/lldb/test/API/lang/cpp/stl/main.cpp:1:10: fatal error: 'cstdio' file not found
    1 | #include <cstdio>
      |          ^~~~~~~~
1 error generated.
make: *** [Makefile.rules:608: main.o] Error 1
make: Leaving directory '/b/s/w/ir/x/w/llvm_build/lldb-test-build.noindex/lang/cpp/stl/TestSTL.test_SBType_template_aspects_dwarf'

Failed bot task: https://ci.chromium.org/ui/p/fuchsia/builders/toolchain.ci/lldb-linux-x64/b8733980216892309505/overview

Looking at the clang invocation, I believe the issue is that it forced the test to be build with GNU libstdc++ , and our bots don't have any GNU toolchain installed.

Is there a reason why LLDB doesn't use LLVM's own libc++?
And if libstdc++ is not present, shouldn't the build system just disable the test that depends on it instead of blindly assume it exists?

[zeroomega]

This will break every test under lldb/test/API/lang/cpp/stl on non darwin machines if GNU libstdc++ is not installed.

I don't think it is a correct to assume the host machine has libstdc++. Plenty of bots are set up in a way to avoid accidentally (and implicitly) introducing GNU toolchains in a LLVM build by removing the entire GNU toolchain from the host machine. It appears the upstream LLDB bots are not set up in this way.

If libstdc++ is required to run this test (and it is the indended behavior), the correct way is to introduce LLVM cmake flag, e.g. LLDB_STDCPP_PATH to explicitly specify the path to the libstdc++, and disable the tests that depends on it in the build system.

But from what I see in the patch history. I believe both tests are not bound to a specific c++ lib. It just need either libc++ or libstdc++. And specifically, if libcxx is linked statically, the test runner somehow needs the libc++ shared object, but it may be not present. If that is the case, the way this patch addresses the test failure on upstream llvm bot is wrong. The correct way it to fix the test runner 1> make sure it builds and works properly when libc++ is linked statically instead of depending on the presence of shared object 2> skip the tests if libcxx and libstd++ is not present instead of forcing using libstdc++ regardless of host environment.

Looking at logics at

llvm-project/lldb/test/API/lang/cpp/stl/TestStdCXXDisassembly.py

Line 34 in 4ddea29

for i in range(target.GetNumModules()):

I think what contributed to the failure on our bot is that it found the libstdc++.so or libc++.so in the modules. Since we don't have gcc installed, the header files for libstdc++ doesn't exist, but clang was invoked with -stdlib=libstdc++ so it is forced to look for gcc's c++ header files. They do not exist. I am not 100% sure if this logic will skip the test if both so files are absent (it looks to me the test will fail with "FAILHORRIBLYHERE"). This logic is also not present in TestSTL.py. So it will fail consistently if none of the so files are present.

[Michael137]

We are seeing test failures on lldb-api :: lang/cpp/stl/TestSTL.py after this patch is landed.

Error when building test subject.

Build Command:
/b/s/w/ir/x/w/cipd/bin/make VPATH=/b/s/w/ir/x/w/llvm-llvm-project/lldb/test/API/lang/cpp/stl -C /b/s/w/ir/x/w/llvm_build/lldb-test-build.noindex/lang/cpp/stl/TestSTL.test_SBType_template_aspects_dwarf -I /b/s/w/ir/x/w/llvm-llvm-project/lldb/test/API/lang/cpp/stl -I /b/s/w/ir/x/w/llvm-llvm-project/lldb/packages/Python/lldbsuite/test/make -f /b/s/w/ir/x/w/llvm-llvm-project/lldb/test/API/lang/cpp/stl/Makefile MAKE_DSYM=NO all ARCH=x86_64 CC=/b/s/w/ir/x/w/cipd/clang/bin/clang CC_TYPE=clang CXX=/b/s/w/ir/x/w/cipd/clang/bin/clang++ LLVM_AR=/b/s/w/ir/x/w/llvm_build/./bin/llvm-ar AR=/b/s/w/ir/x/w/llvm_build/./bin/llvm-ar OBJCOPY=/b/s/w/ir/x/w/llvm_build/./bin/llvm-objcopy STRIP=/b/s/w/ir/x/w/llvm_build/./bin/llvm-strip ARCHIVER=/b/s/w/ir/x/w/llvm_build/./bin/llvm-ar DWP=/b/s/w/ir/x/w/llvm_build/./bin/llvm-dwp CLANG_MODULE_CACHE_DIR=/b/s/w/ir/x/w/llvm_build/lldb-test-build.noindex/module-cache-clang/lldb-api LLDB_OBJ_ROOT=/b/s/w/ir/x/w/llvm_build/tools/lldb OS=Linux HOST_OS=Linux

Build Command Output:
make: Entering directory '/b/s/w/ir/x/w/llvm_build/lldb-test-build.noindex/lang/cpp/stl/TestSTL.test_SBType_template_aspects_dwarf'
/b/s/w/ir/x/w/cipd/clang/bin/clang++  -std=c++11 -g -O0  -m64 -I/b/s/w/ir/x/w/llvm-llvm-project/lldb/packages/Python/lldbsuite/test/make/../../../../..//include -I/b/s/w/ir/x/w/llvm_build/tools/lldb/include -I/b/s/w/ir/x/w/llvm-llvm-project/lldb/test/API/lang/cpp/stl -I/b/s/w/ir/x/w/llvm-llvm-project/lldb/packages/Python/lldbsuite/test/make -include /b/s/w/ir/x/w/llvm-llvm-project/lldb/packages/Python/lldbsuite/test/make/test_common.h -fno-limit-debug-info    -stdlib=libstdc++   --driver-mode=g++ -MT main.o -MD -MP -MF main.d -c -o main.o /b/s/w/ir/x/w/llvm-llvm-project/lldb/test/API/lang/cpp/stl/main.cpp
/b/s/w/ir/x/w/llvm-llvm-project/lldb/test/API/lang/cpp/stl/main.cpp:1:10: fatal error: 'cstdio' file not found
    1 | #include <cstdio>
      |          ^~~~~~~~
1 error generated.
make: *** [Makefile.rules:608: main.o] Error 1
make: Leaving directory '/b/s/w/ir/x/w/llvm_build/lldb-test-build.noindex/lang/cpp/stl/TestSTL.test_SBType_template_aspects_dwarf'

Failed bot task: https://ci.chromium.org/ui/p/fuchsia/builders/toolchain.ci/lldb-linux-x64/b8733980216892309505/overview

@slydiman Lets just revert the Makefile changes in this PR and let the test pick whatever's on the system. The test changes ensure we skip it in the statically linked builds anyway.

Is there a reason why LLDB doesn't use LLVM's own libc++? And if libstdc++ is not present, shouldn't the build system just disable the test that depends on it instead of blindly assume it exists?

We do use a local libc++ build for libc++-specific tests. Otherwise we just use whatever is on the system. I do think the silent fallbacks in some cases could be cleaned up: #106885 (review). Though I'm a bit surprised clang lets you specify -stdlib=libstdc++ but not warn/error out when it's ignored.