incremental compilation: could not create session directory lock file: No locks available (os error 37)

[szignal]

I was unable to incrementally compile using cargo build.

I had added the rand dependency to the toml file, per the advice of the Guessing Game exercise in Rust Programming Language (Second Edition).

Before this particular error, I had just finished setting up a synced folder with NFS on my vagrant box (ubuntu 16.04) where rust was installed. There is some issue using the default vagrant shared folder setup with rust.

I tried this code:

use std::io;

fn main() {
    println!("Guess the number");

    println!("Please input your guess.");

    let mut guess = String::new();

    io::stdin().read_line(&mut guess)
        .expect("Failed to read line");

    print!("You guessed: {}", guess);
}

I expected to see this happen:

compile

Instead, this happened:

    Updating registry `https://github.com/rust-lang/crates.io-index`
 Downloading rand v0.3.22                                                       
 Downloading libc v0.2.40                                                       
 Downloading rand v0.4.2                                                        
   Compiling libc v0.2.40                                                       
   Compiling rand v0.4.2
   Compiling rand v0.3.22
   Compiling guessing_game v0.1.0 (file:///Rust/projects/guessing_game)
error: incremental compilation: could not create session directory lock file: No locks available (os error 37)

thread 'rustc' panicked at 'librustc/session/mod.rs:727: Trying to get session directory from IncrCompSession `NotInitialized`', librustc/session/mod.rs:1180:26
note: Run with `RUST_BACKTRACE=1` for a backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.25.0 (84203cac6 2018-03-25) running on x86_64-unknown-linux-gnu

error: Could not compile `guessing_game`.

rustc --version --verbose:

rustc 1.25.0 (84203cac6 2018-03-25)
binary: rustc
commit-hash: 84203cac67e65ca8640b8392348411098c856985
commit-date: 2018-03-25
host: x86_64-unknown-linux-gnu
release: 1.25.0
LLVM version: 6.0

Backtrace:

thread 'rustc' panicked at 'librustc/session/mod.rs:727: Trying to get session directory from IncrCompSession `NotInitialized`', librustc/session/mod.rs:1180:26
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::_print
             at libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at libstd/sys_common/backtrace.rs:59
             at libstd/panicking.rs:380
   3: std::panicking::default_hook
             at libstd/panicking.rs:396
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:576
   5: std::panicking::begin_panic
   6: rustc::session::opt_span_bug_fmt::{{closure}}
   7: rustc::ty::context::tls::with_opt
   8: rustc::session::opt_span_bug_fmt
   9: rustc::session::bug_fmt
  10: rustc::session::Session::incr_comp_session_dir
  11: rustc_incremental::persist::fs::garbage_collect_session_directories
  12: rustc_driver::driver::phase_2_configure_and_expand_inner::{{closure}}
  13: rustc_driver::driver::phase_2_configure_and_expand_inner
  14: rustc_driver::driver::compile_input
  15: rustc_driver::run_compiler

[glaubitz]

This is an issue we are currently seeing on linux-sparc64 as well [1]:

error: incremental compilation: could not create session directory lock file: No such process (os error 3)

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.24.1 running on sparc64-unknown-linux-gnu

thread 'rustc' panicked at 'src/librustc/session/mod.rs:665: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/session/mod.rs:1141:26
note: Run with `RUST_BACKTRACE=1` for a backtrace.

error: Could not compile `build_helper`.
warning: build failed, waiting for other jobs to finish...

I have not yet figured out where this comes from. The same rust version worked fine in the past.

[1] https://buildd.debian.org/status/fetch.php?pkg=rustc&arch=sparc64&ver=1.25.0%2Bdfsg1-1%7Eexp2&stamp=1523052949&raw=0

CC @jrtc27 @psumbera @mkarcher

[michaelwoerister]

NFS does not support locking files very well, I guess. It would be good if there was some way to make Cargo put the incremental folder somewhere else (into /tmp for example).

cc @rust-lang/cargo

[jrtc27]

Although the original report mentions NFS, we're not using NFS for the sparc64 buildds; the two issues have similar but different error messages ("No locks available" vs "No such process") so they could well be different problems.

[michaelwoerister]

@jrtc27, yes, sounds like a different root cause.

[alexcrichton]

In Cargo we disable file locking whenever NFS is detected, although that may not work well for rustc as it's semi-reasonable to expect Cargo to not be run in parallel much on one system but rustc probably is being executed in parallel by Cargo at least.

An alternative to moving the incremental directory is moving the locks for the incremental directory. I've considered this for Cargo where we actually want to enable file locking on NFS directories. That'd require more intrusive support in rustc, though.

[glaubitz]

If anyone wants to reproduce this problem, there are sparc64 porterboxes available in the gcc compile farm, both running Solaris and Linux. Any open source developer can apply for an account:

https://cfarm.tetaneutral.net/machines/list/

[michaelwoerister]

Is this on a file system that should support locks in general?

An alternative to moving the incremental directory is moving the locks for the incremental directory. I've considered this for Cargo where we actually want to enable file locking on NFS directories. That'd require more intrusive support in rustc, though.

As long as we don't want to support two different machines working on the same NFS share simultaneously (which we don't really do anyway), we could indeed just allocate that lock files in /tmp. The only real restriction for lock files is that they mustn't be contained in the session directory that they protect.

[glaubitz]

Is this on a file system that should support locks in general?

It's a standard off-the-shelf installation of Debian on ext4. Nothing special. There is no NFS or anything involved and it affects sparc64 machines with the same configuration as comparable x86_64 machines.

[michaelwoerister]

Interesting. That's unexpected.

[glaubitz]

You can test it yourself on the Linux/sparc64 porterbox available in the gcc compile farm. It's a rather fast machine.