lld looks for libraries given via relative paths at wrong place (relative to system paths)

[kalibera]

When libraries are passed to the linker as relative paths (e.g. "../../../../lib/libLLVMSupport.a"), they are sometimes looked for relative to the system installation path (e.g. "/something/usr/x86_64-pc-linux-gnu/lib/clang/17/lib/../../../../lib/libLLVMSupport.a"). Instead, they should be looked up always relative to the current working directory.

I've ran into this when cross-compiling a native LLVM compiler (to run on Windows/aarch64 and produce code for Windows/aarch64), using a cross-compiler running on Linux/x86_64. Linking failed due to errors like "ld.lld: error: unknown file type: CommandLine.cpp.o". The correct "libLLVMSupport.a" library has been available relatively to the current working directory in "../../../../lib/libLLVMSupport.a", but the linker instead used an incompatible version (for Linux/x86_64) which happened to be located in "/something/usr/x86_64-pc-linux-gnu/lib/clang/17/lib/../../../../lib/libLLVMSupport.a". When I deleted that incompatible library from that location, the linking passed as it should.

I'm attaching a narrowed-down example based on this problem. On my system, with that "incompatible" library in place,

cd test/sub/sub/sub/sub/
./test1

does

/home/tomas/toolchain/clang/toolchain_libs/mxe/usr/x86_64-pc-linux-gnu/bin/aarch64-w64-mingw32.static.posix-clang++ \
TableGen.cpp.obj \
../../../../lib/libLLVMSupport.a

and outputs

ld.lld: error: unknown file type: CommandLine.cpp.o
ld.lld: error: unknown file type: ManagedStatic.cpp.o
ld.lld: error: unknown file type: PrettyStackTrace.cpp.o
ld.lld: error: unknown file type: raw_ostream.cpp.o
ld.lld: error: unknown file type: Signals.cpp.o
clang: error: linker command failed with exit code 1 (use -v to see invocation) 

With the "incompatible" library not in place, the output includes only complaints about undefined symbols, as it should, because I've narrowed down the example. The archive includes also outputs from strace, which make it clear that lld opens the "incompatible" library at the wrong place:

openat(AT_FDCWD, "/home/tomas/toolchain/clang/toolchain_libs/mxe/usr/x86_64-pc-linux-gnu/lib/clang/17/lib/../../../../lib/libLLVMSupport.a", O_RDONLY|O_CLOEXEC) = 4

The archive also includes the "incompatible" version of the library one could place in the corresponding place to reproduce the problem.

test.zip

@mstorsjo
@tru @rnk @zmodem

[tru]

Is this the COFF or ELF backend? are you producing code for Linux or Windows?

[kalibera]

I am producing code for Windows. The "compatible" library that should be used is COFF. The "incompatible" that is used by accident is ELF.

Per offline communication @mstorsjo knows almost surely why this is happening, I will let him comment.

[tru]

I think I know as well considering it's the COFF backend. I can have a look unless Martin already have a patch.

[mstorsjo]

Here's another even smaller testcase of the same issue. It's a bit more contrieved (one probably wouldn't be doing it like this), but it showcases the issue much clearer:

• lib.c:

void myLibFunc(void) { }

• main.c:

void myLibFunc(void);
void mainCRTStartup() {
	myLibFunc();
}

To run:

$ clang -target x86_64-linux-gnu -c lib.c -o lib-linux.o
$ clang -target aarch64-windows-msvc -c lib.c -o lib-win.o
$ clang -target aarch64-windows-msvc -c main.c -o main-win.o
$ mkdir -p lib-linux
$ rm -f lib-linux/mylib.a mylib.a
$ llvm-ar rcs lib-linux/mylib.a lib-linux.o
$ llvm-ar rcs mylib.a lib-win.o
$ lld-link main-win.o -libpath:lib-linux mylib.a -out:main-win.exe -subsystem:console
# Success; mylib.a resolves to the file in the local directory, not in -libpath
$ cp lib-linux/mylib.a $(dirname $(which lld-link))/../lib
$ lld-link main-win.o -libpath:lib-linux mylib.a -out:main-win.exe -subsystem:console
lld-link: error: unknown file type: lib-linux.o
# Failure; mylib.a is found in $(dirname $(which lld-link))/../lib

The issue can be showed with the new -print-search-paths option to lld. Unfortnately, in this case, it errors out due to the unknown file type before it gets a chance of printing anything. With a modification to print earlier in the process, and a modification to quote the outputs, to make empty entries clearer, I get the following:

$ lld-link main-win.o -libpath:lib-linux mylib.a -out:main-win.exe -subsystem:console -print-search-paths
Library search paths:
  "/home/martin/clang-nightly/lib/clang/17/lib/windows"
  "/home/martin/clang-nightly/lib/clang/17/lib"
  "/home/martin/clang-nightly/lib"
  ""
  "lib-linux"

lld-link: error: unknown file type: lib-linux.o

So the core of the issue, is that the toolchain directories, <toolchain>/lib and <toolchain>/lib/clang/<ver>/lib, are searched before the current directory. Generally, one could think that's reasonable, but:

• If one does a full install of LLVM, i.e. LLVM_INSTALL_TOOLCHAIN_ONLY set to FALSE, then <toolchain>/lib will contain files like libLLVMSupport.a for the host architecture. Accidentally picking up and preferring this file instead of a different one when trying to cross compile is entirely fatal. With regular use of lld-link, the naming style usually doesn't coincide, but in mingw contexts, one can try to link a libLLVMSupport.a for a cross target, but end up including one for the host

• In the case that @kalibera showed, a very unexpected combination happened; a input of ../../../../lib/libLLVMSupport.a happened to match the early search path entry <toolchain>/lib/clang/17/lib resolving into <toolchain>/lib/libLLVMSupport.a.

Suggested change 1

I think that the empty path, i.e. resolving files based on the current directory, always should be first in the search path, before the toolchain files. Before the recent changes, this was always first, before anything grabbed from the windows SDK or from the $LIB env variable. addClangLibSearchPaths uses searchPaths.insert(searchPaths.begin(), with a comment that this is to make sure things are added before whatever is taken from MSVC or the WinSDK. But by doing that, it's also added before the first empty path entry.

I think we could change addClangLibSearchPaths to just append to searchPaths, but restructure the calls to add things in the desired order.

Currently we have this:

  // Construct search path list.
  searchPaths.emplace_back("");
  for (auto *arg : args.filtered(OPT_libpath))
    searchPaths.push_back(arg->getValue());
  detectWinSysRoot(args);
  if (!args.hasArg(OPT_lldignoreenv) && !args.hasArg(OPT_winsysroot))
    addLibSearchPaths();
  addClangLibSearchPaths(argsArr[0]);

But if we really want to have this priority order, if we'd change addClangLibSearchPaths to plain append, we could have this instead:

  // Construct search path list.
  searchPaths.emplace_back("");
  addClangLibSearchPaths(argsArr[0]);
  for (auto *arg : args.filtered(OPT_libpath))
    searchPaths.push_back(arg->getValue());
  detectWinSysRoot(args);
  if (!args.hasArg(OPT_lldignoreenv) && !args.hasArg(OPT_winsysroot))
    addLibSearchPaths();

Suggested change 2

Secondly, b6c2f10 changed so that we do resolve relative paths relative to the libpaths too. This was done with the intent of allowing <triple>/libclang_rt.builtins.a to resolve relative to a libpath. But I think it might be reasonable to not allow this with a relative path with .., what do you think, and only ever resolve such paths relative to the current directory?

Suggested change 3

For mingw contexts, we probably shouldn't be doing any of this logic for auto-adding search paths. In mingw contexts, the linker is always invoked (by the compiler driver) with suitable -L parameters specifying the intended search order. So there, we strictly don't need this logic at all.

I made a patch to do this some months ago, https://reviews.llvm.org/D144084, but when applied, this struck an issue where some compiler-rt test expects to invoke LLD in an MSVC environment, but with -lldmingw to opt in to mingw style handling of weak symbols - this testcase relies on autofinding MSVC libraries while enabling mingw mode. So we probably should separate these concerns a bit, either add a separate flag for -lldmingw-style-weak-symbols, or make a new flag -lld-no-auto-libpaths, which the lld mingw frontend always passes in.

I just haven't had a real demand for going forward with this patch so I haven't quite had time, but I guess this is a good case for it.

[JiaT75]

Hello!

So the core of the issue, is that the toolchain directories, <toolchain>/lib and <toolchain>/lib/clang/<ver>/lib, are searched before the current directory. Generally, one could think that's reasonable, but:

I also ran into this bug a few days ago and have been investigated it a bit. I was planning on submitting a bug report today but you beat me to it. You did a better job than I would have writing it up so its good that you did :)
I believe this is serious problem since it can lead to unexpected results when linking.

I see in the release notes for 17.0.1:
"By default lld-link will now search for libraries in the toolchain directories. Specifically it will search: /lib, /lib/clang//lib and /lib/clang//lib/windows."

In my case, my CI toolchain caught the bug in 17.0.1. I am building liblzma.a and disabling various features and testing that the build and tests are successful. The problem is that on MSYS2 CLANG64 environment liblzma.a is installed in the system libraries (/clang64/lib/). So when building the xz.exe command line tool, it links with the system liblzma.a instead of the one in the local directory.

This causes an obvious problem if trying to build and distribute a new version of a program that depends on a library built locally.

So it makes sense why this issue did not appear until 17.0.1. The expected behavior should be to check the local directory first, then the toolchain directories. I believe this problem is only with the COFF backend since I could not reproduce it on my Ubuntu machine after upgrading to the 17.0.1 release. It possible I set up my test wrong so don't believe me 100%.

[tru]

Secondly, b6c2f10 changed so that we do resolve relative paths relative to the libpaths too. This was done with the intent of allowing /libclang_rt.builtins.a to resolve relative to a libpath. But I think it might be reasonable to not allow this with a relative path with .., what do you think, and only ever resolve such paths relative to the current directory?

I think that's very reasonable and I would think that change would be easy and good.

[mstorsjo]

See #67856, #67857 and #67858 for one set of potential fixes for this issue.

[mstorsjo]

I forgot to mention here, that this issue actually reproduces quite easily - if building LLVM with the CMake Makefile generator, as opposed to Ninja. With Ninja, all the build commands are executed in the same directory, the base of the build directory. However, with the makefile generator, it uses recursive makefiles, thus it enters subdirectories and executes commands there.

That's why there are relative path references such as ../../../../lib/libLLVMSupport.a. And such a path, applied on <toolchain>/lib/clang/18/lib easily finds <toolchain>/lib/libLLVMSupport.a.