Unreadable error messages on linking failure

[fschutt]

This is a very annoying aspect of rustc - if it fails to link a library, it doesn't tell you the path, it doesn't tell you which library it couldn't find, it just spits out a completely unreadable error message and quits. For example:

   Compiling deflate v0.7.17
error: could not exec the linker `cc`: No such file or directory (os error 2)
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/home/felix/.rustup/toolchains/stable-
x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" 
"/home/felix/Development/srtmtoimage/target/debug/build/kernel32-sys-
23866d7aeb753806/build_script_build-23866d7aeb753806.build_script_build0.rust-cgu.o" "-o" 
"/home/felix/Development/srtmtoimage/target/debug/build/kernel32-sys-
23866d7aeb753806/build_script_build-23866d7aeb753806" 
"/home/felix/Development/srtmtoimage/target/debug/build/kernel32-sys-
23866d7aeb753806/build_script_build-23866d7aeb753806.crate.allocator.rust-cgu.o" "-Wl,--gc-
sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs" "-L" 
"/home/felix/Development/srtmtoimage/target/debug/deps" "-L" "/home/felix/.rustup/toolchains/stable-
x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" 
"/home/felix/Development/srtmtoimage/target/debug/deps/libbuild-055e23f8aa405a7b.rlib" 
"/home/felix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-
gnu/lib/libstd-fe0b1b991511fcaa.rlib" "/home/felix/.rustup/toolchains/stable-x86_64-unknown-linux-
gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librand-3d7b10e850a67e89.rlib" 
"/home/felix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-
gnu/lib/liballoc_jemalloc-28484309357fd6f1.rlib" "/home/felix/.rustup/toolchains/stable-x86_64-
unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_system-751808ba756769d5.rlib"
 "/home/felix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-
gnu/lib/libpanic_unwind-8cb97051d8238386.rlib" "/home/felix/.rustup/toolchains/stable-x86_64-
unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-25cc9b024a02d330.rlib"
 "/home/felix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-
gnu/lib/liblibc-d42e80cee81b06ce.rlib" "/home/felix/.rustup/toolchains/stable-x86_64-unknown-linux-
gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-78c21267a2dc15c1.rlib" 
"/home/felix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-
gnu/lib/libstd_unicode-0e1b544c94586415.rlib" "/home/felix/.rustup/toolchains/stable-x86_64
-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-0c5e3d6c117f8c44.rlib"
 "/home/felix/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-
gnu/lib/libcompiler_builtins-bd7cc5ada1e908e0.rlib" "-Wl,-Bdynamic" "-l" "dl" "-l" "rt" "-l" "pthread" "-l"
 "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "pthread" "-l" "util"

error: aborting due to previous error

error: Could not compile `rayon-core`.
warning: build failed, waiting for other jobs to finish...
error: Could not compile `kernel32-sys`.
warning: build failed, waiting for other jobs to finish...
error: build failed

Right, now try and read what the actual error is. In this case, the linker cc is missing, but the error message is similar if a system library is missing - ex. you tried to use curl-rs but you don't have libcurl installed. In that case, you have to search for the missing library somewhere in the last arguments.

Please:

• Don't just quit with "No such file or directory" - it's one of the most useless error messages ever. Please include at least the file name where rustc expected the linker / library to be.
• If it is a linker error, please don't output the whole paths to the libraries. Just tell me which library wasn't found
• Maybe include the folders that rust searched for the libraries (helpful for debugging $PATH issues)
• Don't output the location of every object file / library, it's not helpful.

That would be my suggestion for improving cargo. It's simply annoying to figure out which library is missing (99,99% it's a system dependency).

Linked from rust-lang/cargo#4863

[estebank]

The problem here is that the error you're seeing is not coming from rustc, but rather from shelling out and trying to run cc. We can improve this error in particular by indeed verifying that the linker we're trying to run is there and emitting a different error for it in link.rs and write.rs, including the path information from the environment.

As further improvement, rustc could also probably scrape the output of the linker for known errors to minimize the output in those cases, and fall back to the current behavior (printing the entirety of the linkers output).

[retep998]

There's two typical kinds of linking failure 1. the linker does not exist 2. the linker does exist but failed to link successfully.

In the former case we definitely should not print the linker command line because it's meaningless when we couldn't find a linker to begin with. Instead printing out some diagnostic information about where rustc tried to look for the linker would be more useful (in particular this would be very useful for msvc where finding the linker is actually really complicated).

In the latter case the linker command line is important as it contains all the information needed to debug issues due to the linker failing to succeed. If you want to know what folders were searched for libraries by the linker you just look for the relevant -L or /LIBPATH flags. Oftentimes there is a lot of irrelevant junk in the command line but trying to filter it down is a really difficult challenge because sometimes the stuff that seems like junk contains some vital detail.

[fschutt]

@retep998 @estebank If this is an actual bug, I'd love to do this as my first contribution to the rust compiler. I'm not the best Rust developer, but I'm not exactly new to the language either. So my question would be:

• Can we agree that this is undesired behaviour and should be fixed?
• How would you think the output should look like (i.e. what should the error message tell)? I'd personally agree with @retep998:
  • If the linker doesn't exist, the linker arguments are not printed, because it wouldn't make sense, instead the paths where the linker was searched for are printed in some way.
  • If the linker exists, but failed to build, we need to somehow identify the library that could not be linked and the paths the library was searched for.
• I don't know exactly how the error handling in rustc works - are errors handled with abstractions (error_chain, failure, etc.) or does the compiler just panic / abort in place?

Does rustc already know the name of the library it's linking?Theoretically speaking, the strategy I'd think about: If rustc invokes the linker with let's say -lcurl it has to give the string "curl" somehow to the linker right? So what the compiler could do is to track what the latest linker argument was - and if the linker throws an error, we parse the output, print the paths and exit.

So for case 1:

error: could not exec the linker `cc`: linker not found (os error 2)
  |
  = note: this error originates in the crate "deflate"
  = note: searched for linker in the following paths:
     /usr/bin/
     /usr/local/bin
     /usr/sbin

and for case 2:

error: linking with `cc` failed: exit code: 1
 --> deflate/build.rs:2:5
  |
2 |     println!("cargo:rustc-link-lib=asdfasdf");
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
= note: /usr/bin/ld: cannot find library "asdfasdf"
= note: searched for library in the following paths:
     /usr/bin/
     /usr/local/bin
     /usr/sbin
= note: collect2: error: ld returned 1 exit status

[retep998]

Trying to tell which paths were searched for the library is going to be really hard because rustc does not actually know all the paths where the linker searches for libraries. Environment variables and system configuration can determine additional directories where the linker will search which rustc never told the linker to search in, and this differs between different toolchains (msvc again being significantly different in this regard).

There is one linker output that I know we can reliably parse and use to print out a much more informative message. If we're targeting pc-windows-msvc and the linker output contains link: extra operand then that is definitely not the linker we want but the coreutils command link, in which case rustc needs to print out a very helpful message telling the user they need to install the VC++ build tools.

[estebank]

I'd love to do this as my first contribution to the rust compiler. I'm not the best Rust developer, but I'm not exactly new to the language either.

That is great!

• Can we agree that this is undesired behaviour and should be fixed?

Yes, I believe we can improve significantly over the current output.

• How would you think the output should look like (i.e. what should the error message tell)? I'd personally agree with @retep998:
  • If the linker doesn't exist, the linker arguments are not printed, because it wouldn't make sense, instead the paths where the linker was searched for are printed in some way.

I definitely believe this is the main fix for this issue.

• If the linker exists, but failed to build, we need to somehow identify the library that could not be linked and the paths the library was searched for.

We already output the linker's result, which should be enough but not great. Scraping the output in order to output the error in a consistent way as the rest of rustc errors would be the best bet, if not exhaustive.

• I don't know exactly how the error handling in rustc works - are errors handled with abstractions (error_chain, failure, etc.) or does the compiler just panic / abort in place?

It depends, on cases like this the compiler usually bails out. We have a struct DiagnosticBuilder which encodes the errors you normally see, which can have a message, an error code, a bunch of spans with labels, and a bunch of note/help with optional span information. The proposed output

error: could not exec the linker `cc`: linker not found (os error 2)
  |
  = note: this error originates in the crate "deflate"
  = note: searched for linker in the following paths:
          /usr/bin/
          /usr/local/bin
          /usr/sbin

would be created by an let mut err: DiagnosticBuilder with err.note(&format!("this error originates in the crate \"{}\"", crate_name)); and err.note("searched for linker in the following paths:\n/usr/bin/\n/usr/local/bin\n/usr/sbin");.

Does rustc already know the name of the library it's linking? Theoretically speaking, the strategy I'd think about: If rustc invokes the linker with let's say -lcurl it has to give the string "curl" somehow to the linker right? So what the compiler could do is to track what the latest linker argument was - and if the linker throws an error, we parse the output, print the paths and exit.

I believe it should know that, but in this case where cc is not available, the crate failing to compile is not as relevant, and will be pointed out right before:

   Compiling deflate v0.7.17
error: could not exec the linker `cc`: linker not found (os error 2)
  |
  = note: searched for linker in the following paths:
          /usr/bin/
          /usr/local/bin
          /usr/sbin

and for case 2:

error: linking with `cc` failed: exit code: 1
 --> deflate/build.rs:2:5
  |
2 |     println!("cargo:rustc-link-lib=asdfasdf");
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
= note: /usr/bin/ld: cannot find library "asdfasdf"
= note: searched for library in the following paths:
     /usr/bin/
     /usr/local/bin
     /usr/sbin
= note: collect2: error: ld returned 1 exit status

I believe this is gonna be trickier to accomplish (would have to dig deeper in the code to see if it is even possible at the moment), I would keep that feature as a follow up PR, so that you can focus on a single smaller task to get a feel for the compiler codebase. Once you're acquainted with it, we can look to see if we can get the relevant span (I'm not sure if it's possible), how to expose the missing library name and the search paths.

How does that sound, @fschutt?

[fschutt]

Sure, doing the easy things first. The thing I'm currently running into is this:

info: looks like you are running this command under `sudo`
      and so in order to preserve your $HOME this will now
      use vendored sources by default. Note that if this
      does not work you should run a normal build first
      before running a command like `sudo make install`
Updating submodules
error: failed to load source for a dependency on `cc`

Caused by:
  Unable to update registry `https://github.com/rust-lang/crates.io-index`

Caused by:
  failed to update replaced source registry `https://github.com/rust-lang/crates.io-index`

Caused by:
  failed to read root of directory source: ~/rust/src/vendor

Caused by:
  No such file or directory (os error 2)

I am not sure what "you should run a normal build first" means - I just ran sudo ./x.py build. Not sure what "normal build" means I should do. I cloned into ~/rust, but the folder ~/rust/src/vendor doesn't exist. I tried configure + make, but that seems to do the same. This happens even with a clean install.

[estebank]

Why are you running with sudo? The build should work as expected running as your user (./x.py build).

[fschutt]

Thanks, I got it to build. I'll try to do a PR.

[fschutt]

So I now changed it to output:

error: could not exec the linker `cc`: linker not found: No such file or directory (os error 2)

error: aborting due to previous error

... however, I think it's easier shortening the error message to:

error: linker `cc` not found: No such file or directory (os error 2)

error: aborting due to previous error

It's better to read imho, because if you use a terminal, you need less time to understand the error.

[estebank]

Could you move the OS error to a note? That way people the error message itself is short and to the point, while the extra context still available to the user.

error: linker `cc` not found
  |
  = note: No such file or directory (os error 2)

It would also be amazing if depending on the environment being run we had an error message with a targeted description similar to the troubleshooting in the first edition of The Book telling you how to fix the problem without having to google it.

Nitpicks aside, feel free to post the PR at your earliest convenience! I love seeing this kind of small, incremental but dramatic usability improvements :D

[fschutt]

@estebank See #47052