Can't link libomptarget.so.16 because of missing symbols

[liam-clink]

When building LLVM 16.0.0 with Clang 15.0.3 using config:

cmake -S ../llvm -B . -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_ENABLE_PROJECTS="bolt;llvm;clang;lld;lldb;clang-tools-extra;cross-project-tests;libclc;polly;pstl" -DLLVM_ENABLE_RUNTIMES="openmp;libcxx;libcxxabi;libunwind;compiler-rt" -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DLIBCXX_USE_COMPILER_RT=YES -DLLVM_USE_LINKER=lld -DLIBCXXABI_USE_LLVM_UNWINDER=YES -DLLVM_INSTALL_UTILS=ON -DCMAKE_INSTALL_PREFIX=$HOME/llvm15_temp -DLLDB_ENABLE_PYTHON=TRUE -DPython3_EXECUTABLE=`which python3` -DLLVM_PARALLEL_LINK_JOBS=1 -DLLVM_ENABLE_LIBCXX=ON

I get an error at the linking stage of libomptarget.so.16

Linking CXX shared library /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libomptarget.so.16

ld.lld: error: undefined symbol: llvm::sys::DynamicLibrary::getPermanentLibrary(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*)
>>> referenced by ompt_callback.cpp
>>>               openmp/libomptarget/src/CMakeFiles/omptarget.dir/ompt_callback.cpp.o:(OmptLibraryConnectorTy::initialize())
>>> referenced by rtl.cpp
>>>               openmp/libomptarget/src/CMakeFiles/omptarget.dir/rtl.cpp.o:(RTLsTy::attemptLoadRTL(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, RTLInfoTy&))

ld.lld: error: undefined symbol: std::__1::error_category::~error_category()
>>> referenced by Error.cpp
>>>               Error.cpp.o:(llvm::ErrorList::convertToErrorCode() const) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by Error.cpp
>>>               Error.cpp.o:((anonymous namespace)::ErrorErrorCategory::~ErrorErrorCategory()) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by Error.cpp
>>>               Error.cpp.o:(vtable for (anonymous namespace)::ErrorErrorCategory) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced 3 more times

ld.lld: error: undefined symbol: std::__1::error_code::message() const
>>> referenced by Error.cpp
>>>               Error.cpp.o:(llvm::StringError::log(llvm::raw_ostream&) const) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by Error.cpp
>>>               Error.cpp.o:(llvm::ECError::log(llvm::raw_ostream&) const) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by raw_ostream.cpp
>>>               raw_ostream.cpp.o:(llvm::raw_fd_ostream::~raw_fd_ostream()) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a

ld.lld: error: undefined symbol: std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> std::__1::operator+<char, std::__1::char_traits<char>, std::__1::allocator<char>>(char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)
>>> referenced by Error.cpp
>>>               Error.cpp.o:(llvm::StringError::log(llvm::raw_ostream&) const) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by TimeProfiler.cpp
>>>               TimeProfiler.cpp.o:(llvm::timeTraceProfilerWrite(llvm::StringRef, llvm::StringRef)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by TimeProfiler.cpp
>>>               TimeProfiler.cpp.o:(llvm::TimeTraceProfiler::write(llvm::raw_pwrite_stream&)::'lambda'()::operator()() const) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a

ld.lld: error: undefined symbol: std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::basic_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)
>>> referenced by Error.cpp
>>>               Error.cpp.o:(llvm::ErrorInfoBase::message() const) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by TimeProfiler.cpp
>>>               TimeProfiler.cpp.o:(llvm::TimeTraceProfiler::begin(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, llvm::function_ref<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> ()>)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by TimeProfiler.cpp
>>>               TimeProfiler.cpp.o:(llvm::TimeTraceProfiler::begin(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, llvm::function_ref<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> ()>)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced 12 more times

ld.lld: error: undefined symbol: std::__1::error_category::default_error_condition(int) const
>>> referenced by Error.cpp
>>>               Error.cpp.o:(vtable for (anonymous namespace)::ErrorErrorCategory) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by Error.cpp
>>>               Error.cpp.o:(vtable for (anonymous namespace)::_object_error_category) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMObject.a

ld.lld: error: undefined symbol: std::__1::error_category::equivalent(int, std::__1::error_condition const&) const
>>> referenced by Error.cpp
>>>               Error.cpp.o:(vtable for (anonymous namespace)::ErrorErrorCategory) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by Error.cpp
>>>               Error.cpp.o:(vtable for (anonymous namespace)::_object_error_category) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMObject.a

ld.lld: error: undefined symbol: std::__1::error_category::equivalent(std::__1::error_code const&, int) const
>>> referenced by Error.cpp
>>>               Error.cpp.o:(vtable for (anonymous namespace)::ErrorErrorCategory) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by Error.cpp
>>>               Error.cpp.o:(vtable for (anonymous namespace)::_object_error_category) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMObject.a

ld.lld: error: undefined symbol: std::__1::mutex::lock()
>>> referenced by ErrorHandling.cpp
>>>               ErrorHandling.cpp.o:(llvm::report_fatal_error(llvm::Twine const&, bool)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by ErrorHandling.cpp
>>>               ErrorHandling.cpp.o:(llvm::report_bad_alloc_error(char const*, bool)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by TimeProfiler.cpp
>>>               TimeProfiler.cpp.o:(llvm::timeTraceProfilerCleanup()) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced 2 more times

ld.lld: error: undefined symbol: std::__1::mutex::unlock()
>>> referenced by ErrorHandling.cpp
>>>               ErrorHandling.cpp.o:(llvm::report_fatal_error(llvm::Twine const&, bool)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by ErrorHandling.cpp
>>>               ErrorHandling.cpp.o:(llvm::report_bad_alloc_error(char const*, bool)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by TimeProfiler.cpp
>>>               TimeProfiler.cpp.o:(llvm::timeTraceProfilerCleanup()) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced 2 more times

ld.lld: error: undefined symbol: std::__1::recursive_mutex::lock()
>>> referenced by ManagedStatic.cpp
>>>               ManagedStatic.cpp.o:(llvm::ManagedStaticBase::RegisterManagedStatic(void* (*)(), void (*)(void*)) const) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a

ld.lld: error: undefined symbol: std::__1::recursive_mutex::unlock()
>>> referenced by ManagedStatic.cpp
>>>               ManagedStatic.cpp.o:(llvm::ManagedStaticBase::RegisterManagedStatic(void* (*)(), void (*)(void*)) const) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a

ld.lld: error: undefined symbol: std::__1::recursive_mutex::recursive_mutex()
>>> referenced by ManagedStatic.cpp
>>>               ManagedStatic.cpp.o:(llvm::ManagedStaticBase::RegisterManagedStatic(void* (*)(), void (*)(void*)) const) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a

ld.lld: error: undefined symbol: std::__1::recursive_mutex::~recursive_mutex()
>>> referenced by ManagedStatic.cpp
>>>               ManagedStatic.cpp.o:(llvm::ManagedStaticBase::RegisterManagedStatic(void* (*)(), void (*)(void*)) const) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a

ld.lld: error: undefined symbol: std::__1::to_string(unsigned long)
>>> referenced by SmallVector.cpp
>>>               SmallVector.cpp.o:(report_size_overflow(unsigned long, unsigned long)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by SmallVector.cpp
>>>               SmallVector.cpp.o:(report_size_overflow(unsigned long, unsigned long)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by SmallVector.cpp
>>>               SmallVector.cpp.o:(report_at_maximum_capacity(unsigned long)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a

ld.lld: error: undefined symbol: std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::~basic_string()
>>> referenced by SmallVector.cpp
>>>               SmallVector.cpp.o:(report_size_overflow(unsigned long, unsigned long)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by SmallVector.cpp
>>>               SmallVector.cpp.o:(report_size_overflow(unsigned long, unsigned long)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by SmallVector.cpp
>>>               SmallVector.cpp.o:(report_size_overflow(unsigned long, unsigned long)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced 3 more times

ld.lld: error: undefined symbol: std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::append(char const*)
>>> referenced by SmallVector.cpp
>>>               SmallVector.cpp.o:(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> std::__1::operator+[abi:v15003]<char, std::__1::char_traits<char>, std::__1::allocator<char>>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&&, char const*)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by TimeProfiler.cpp
>>>               TimeProfiler.cpp.o:(llvm::timeTraceProfilerWrite(llvm::StringRef, llvm::StringRef)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a

ld.lld: error: undefined symbol: std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::append(char const*, unsigned long)
>>> referenced by SmallVector.cpp
>>>               SmallVector.cpp.o:(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> std::__1::operator+[abi:v15003]<char, std::__1::char_traits<char>, std::__1::allocator<char>>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&&)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a
>>> referenced by raw_ostream.cpp
>>>               raw_ostream.cpp.o:(llvm::raw_string_ostream::write_impl(char const*, unsigned long)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a

ld.lld: error: undefined symbol: std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>::insert(unsigned long, char const*)
>>> referenced by SmallVector.cpp
>>>               SmallVector.cpp.o:(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> std::__1::operator+[abi:v15003]<char, std::__1::char_traits<char>, std::__1::allocator<char>>(char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&&)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a

ld.lld: error: undefined symbol: std::__1::chrono::system_clock::now()
>>> referenced by TimeProfiler.cpp
>>>               TimeProfiler.cpp.o:(llvm::TimeTraceProfiler::TimeTraceProfiler(unsigned int, llvm::StringRef)) in archive /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a

[llvmbot]

@llvm/issue-subscribers-openmp

[MaskRay]

I have tried the CMake command and ninja runtimes can build lib/libomptarget.so.16 (release/16.x branch) or lib/libomptarget.so.17 (main branch) without an issue.

I think you likely mismatch libstdc++ and libc++.

undefined symbol: std::__1::recursive_mutex::lock() indicates a libc++ symbol name while std::__cxx11::basic_string suggests a libstdc++ symbol name. You may want to check whether your clang 15.0.3 is configured correctly.

[liam-clink]

How would I check that it is correctly configured? Like how do I make sure that clang uses lib++?

[liam-clink]

I built the clang using

cmake -S ../llvm -B . -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_ENABLE_PROJECTS="bolt;llvm;clang;lld;lldb;clang-tools-extra;cross-project-tests;libclc;polly;pstl" -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi;libunwind;compiler-rt" -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DLIBCXX_USE_COMPILER_RT=YES -DLLVM_USE_LINKER=lld -DLIBCXXABI_USE_LLVM_UNWINDER=YES -DLLVM_INSTALL_UTILS=ON -DCMAKE_INSTALL_PREFIX=$HOME/llvm15_temp -DLLDB_ENABLE_PYTHON=TRUE -DPython3_EXECUTABLE=`which python3` -DLLVM_PARALLEL_LINK_JOBS=1 -DLLVM_ENABLE_LIBCXX=ON

I specifically am telling clang to use libc++, why would it be trying to use libstdc++?

[jhuber6]

ld.lld: error: undefined symbol: llvm::sys::DynamicLibrary::getPermanentLibrary(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*)
>>> referenced by ompt_callback.cpp
>>>               openmp/libomptarget/src/CMakeFiles/omptarget.dir/ompt_callback.cpp.o:(OmptLibraryConnectorTy::initialize())
>>> referenced by rtl.cpp
>>>               openmp/libomptarget/src/CMakeFiles/omptarget.dir/rtl.cpp.o:(RTLsTy::attemptLoadRTL(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, RTLInfoTy&))

This part is confusing, that file should be provided by LLVM's LibLLVMSupport library. I'm not sure why it's failing to be linked here.

[liam-clink]

You mean the pre-existing one, or the just-built one?

[jhuber6]

It should be pulling it form the current LLVM build since it's added via add_llvm_library. I'm unsure why it could fail to pick those libraries up unfortunately. Do you think you could get a verbose print-out of the command that CMake is using to link / compile that file?

[liam-clink]

Is this what you mean?

[31/103] Linking CXX shared library /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libomptarget.so.16
FAILED: /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libomptarget.so.16 
: && /home/liam/Downloads/llvm-project-16.0.0.src/build/./bin/clang++ --target=x86_64-unknown-linux-gnu -fPIC -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Wall -Wcast-qual -Wformat-pedantic -Wimplicit-fallthrough -Wsign-compare -Wno-enum-constexpr-conversion -Wno-extra -Wno-pedantic -std=c++17 -O3 -DNDEBUG  -Wl,-z,defs -Wl,-z,nodelete -fuse-ld=lld -Wl,--color-diagnostics   -Wl,--gc-sections -shared -Wl,-soname,libomptarget.so.16 -o /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libomptarget.so.16 openmp/libomptarget/src/CMakeFiles/omptarget.dir/api.cpp.o openmp/libomptarget/src/CMakeFiles/omptarget.dir/device.cpp.o openmp/libomptarget/src/CMakeFiles/omptarget.dir/interface.cpp.o openmp/libomptarget/src/CMakeFiles/omptarget.dir/interop.cpp.o openmp/libomptarget/src/CMakeFiles/omptarget.dir/omptarget.cpp.o openmp/libomptarget/src/CMakeFiles/omptarget.dir/ompt_callback.cpp.o openmp/libomptarget/src/CMakeFiles/omptarget.dir/rtl.cpp.o openmp/libomptarget/src/CMakeFiles/omptarget.dir/LegacyAPI.cpp.o  -Wl,-rpath,"\$ORIGIN:/home/liam/Downloads/llvm-project-16.0.0.src/build/runtimes/runtimes-bins/openmp/libomptarget/src/..:"  -Wl,--version-script=/home/liam/Downloads/llvm-project-16.0.0.src/openmp/libomptarget/src/exports  /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a  /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMObject.a  /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMIRReader.a  /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMBitReader.a  /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMAsmParser.a  /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMCore.a  /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMRemarks.a  /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMBitstreamReader.a  /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMMCParser.a  /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMMC.a  /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMDebugInfoCodeView.a  /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMTextAPI.a  /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMBinaryFormat.a  /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMTargetParser.a  /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMSupport.a  -lrt  -ldl  -lm  /usr/lib/x86_64-linux-gnu/libz.so  /usr/lib/x86_64-linux-gnu/libtinfo.so  /home/liam/Downloads/llvm-project-16.0.0.src/build/lib/libLLVMDemangle.a && :

[jhuber6]

It's definitely linking in libLLVMSupport.a (it's actually being linked twice). So I'm not sure why you are getting linker errors for getPermanentLibrary. If you do something like llvm-nm libLLVMSupport.a | grep 'getPermanentLibrary' you should see that symbol, right? That's the part that's confusing me.