Doctests doesn't print the assert failure output

[farodin91]

I run the cargo test command on nightly did not get any output on failing assert_eq!() in doctests.

I expected to see this happen:
Output on stable(1.26)

`rustdoc --test ./predicates-rs/src/lib.rs --crate-name predicates -L dependency=./predicates-rs/target/debug/deps -L dependency=./predicates-rs/target/debug/deps --cfg 'feature="default"' --cfg 'feature="regex"' --cfg 'feature="float-cmp"' --cfg 'feature="difference"' --extern float_cmp=./predicates-rs/target/debug/deps/libfloat_cmp-4a42d8413dc12365.rlib --extern regex=./predicates-rs/target/debug/deps/libregex-30b85987be956602.rlib --extern predicates=./predicates-rs/target/debug/deps/libpredicates-f9ff1b22f35b575a.rlib --extern difference=./predicates-rs/target/debug/deps/libdifference-ed04b642d361199c.rlib`

running 42 tests
test src/boolean.rs - boolean::PredicateBooleanExt::not (line 218) ... ok
test src/boolean.rs - boolean::PredicateBooleanExt::and (line 178) ... ok
test src/boolean.rs - boolean::PredicateBooleanExt::or (line 197) ... ok
test src/boxed.rs - boxed::PredicateBoxExt::boxed (line 80) ... ok
test src/float/close.rs - float::close::IsClosePredicate::distance (line 35) ... ok
test src/constant.rs - constant::always (line 41) ... ok
test src/float/close.rs - float::close::IsClosePredicate::epsilon (line 54) ... ok
test src/constant.rs - constant::never (line 62) ... ok
test src/float/close.rs - float::close::IsClosePredicate::ulps (line 71) ... ok
test src/function.rs - function::FnPredicate<F, T>::fn_name (line 36) ... ok
test src/float/close.rs - float::close::is_close (line 104) ... ok
test src/function.rs - function::function (line 78) ... ok
test src/iter.rs - iter::in_iter (line 100) ... ok
test src/iter.rs - iter::InPredicate<T>::sort (line 49) ... ok
test src/iter.rs - iter::in_hash (line 200) ... ok
test src/lib.rs -  (line 34) ... ok
test src/name.rs - name::PredicateNameExt::name (line 59) ... ok
test src/ord.rs - ord::ge (line 215) ... ok
test src/ord.rs - ord::eq (line 70) ... ok
test src/ord.rs - ord::gt (line 238) ... ok
test src/ord.rs - ord::le (line 192) ... ok
test src/ord.rs - ord::lt (line 170) ... ok
test src/ord.rs - ord::ne (line 92) ... ok
test src/path/existence.rs - path::existence::exists (line 38) ... ok
test src/path/existence.rs - path::existence::missing (line 53) ... ok
test src/path/ft.rs - path::ft::is_dir (line 107) ... ok
test src/path/ft.rs - path::ft::is_file (line 87) ... ok
test src/path/ft.rs - path::ft::is_symlink (line 127) ... ok
test src/str/adapters.rs - str::adapters::PredicateSliceExt::from_file_path (line 169) ... FAILED
test src/str/adapters.rs - str::adapters::PredicateStrExt::trim (line 123) ... ok
test src/str/adapters.rs - str::adapters::PredicateStrExt::from_utf8 (line 138) ... ok
test src/str/basics.rs - str::basics::ContainsPredicate::count (line 139) ... ok
test src/str/basics.rs - str::basics::is_empty (line 35) ... ok
test src/str/basics.rs - str::basics::contains (line 191) ... ok
test src/str/basics.rs - str::basics::ends_with (line 110) ... ok
test src/str/basics.rs - str::basics::starts_with (line 70) ... ok
test src/str/difference.rs - str::difference::DifferencePredicate::distance (line 74) ... ok
test src/str/difference.rs - str::difference::DifferencePredicate::split (line 54) ... ok
test src/str/difference.rs - str::difference::diff (line 108) ... ok
test src/str/difference.rs - str::difference::similar (line 131) ... ok
test src/str/regex.rs - str::regex::is_match (line 80) ... ok
test src/str/regex.rs - str::regex::RegexPredicate::count (line 31) ... ok

failures:

---- src/str/adapters.rs - str::adapters::PredicateSliceExt::from_file_path (line 169) stdout ----
        thread 'rustc' panicked at 'test executable failed:

thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `false`,
 right: `true`', src/str/adapters.rs:8:1
note: Run with `RUST_BACKTRACE=1` for a backtrace.

', librustdoc/test.rs:341:17
note: Run with `RUST_BACKTRACE=1` for a backtrace.


failures:
    src/str/adapters.rs - str::adapters::PredicateSliceExt::from_file_path (line 169)

Instead, this happened:
Output on nightly (rustc 1.28.0-nightly (5bf68db 2018-05-28))

rustdoc --test ./predicates-rs/src/lib.rs --crate-name predicates -L dependency=./predicates-rs/target/debug/deps -L dependency=./predicates-rs/target/debug/deps --cfg 'feature="float-cmp"' --cfg 'feature="default"' --cfg 'feature="regex"' --cfg 'feature="difference"' --extern difference=./predicates-rs/target/debug/deps/libdifference-cf1e9088133b4a68.rlib --extern float_cmp=./predicates-rs/target/debug/deps/libfloat_cmp-0d3ec5c6568f6886.rlib --extern predicates=./predicates-rs/target/debug/deps/libpredicates-6e0d49cf47babc04.rlib --extern regex=./predicates-rs/target/debug/deps/libregex-5aa34502a8174bb1.rlib

running 42 tests
test src/boolean.rs - boolean::PredicateBooleanExt::not (line 218) ... ok
test src/boolean.rs - boolean::PredicateBooleanExt::or (line 197) ... ok
test src/boolean.rs - boolean::PredicateBooleanExt::and (line 178) ... ok
test src/boxed.rs - boxed::PredicateBoxExt::boxed (line 80) ... ok
test src/float/close.rs - float::close::IsClosePredicate::distance (line 35) ... ok
test src/float/close.rs - float::close::IsClosePredicate::epsilon (line 54) ... ok
test src/constant.rs - constant::always (line 41) ... ok
test src/constant.rs - constant::never (line 62) ... ok
test src/float/close.rs - float::close::IsClosePredicate::ulps (line 71) ... ok
test src/function.rs - function::FnPredicate<F, T>::fn_name (line 36) ... ok
test src/float/close.rs - float::close::is_close (line 104) ... ok
test src/function.rs - function::function (line 78) ... ok
test src/iter.rs - iter::InPredicate<T>::sort (line 49) ... ok
test src/iter.rs - iter::in_hash (line 200) ... ok
test src/iter.rs - iter::in_iter (line 100) ... ok
test src/name.rs - name::PredicateNameExt::name (line 59) ... ok
test src/lib.rs -  (line 34) ... ok
test src/ord.rs - ord::eq (line 70) ... ok
test src/ord.rs - ord::ge (line 215) ... ok
test src/ord.rs - ord::gt (line 238) ... ok
test src/ord.rs - ord::le (line 192) ... ok
test src/ord.rs - ord::lt (line 170) ... ok
test src/ord.rs - ord::ne (line 92) ... ok
test src/path/existence.rs - path::existence::exists (line 38) ... ok
test src/path/existence.rs - path::existence::missing (line 53) ... ok
test src/path/ft.rs - path::ft::is_dir (line 107) ... ok
test src/path/ft.rs - path::ft::is_file (line 87) ... ok
test src/path/ft.rs - path::ft::is_symlink (line 127) ... ok
test src/str/adapters.rs - str::adapters::PredicateSliceExt::from_file_path (line 169) ... FAILED
test src/str/adapters.rs - str::adapters::PredicateStrExt::from_utf8 (line 138) ... ok
test src/str/adapters.rs - str::adapters::PredicateStrExt::trim (line 123) ... ok
test src/str/basics.rs - str::basics::ContainsPredicate::count (line 139) ... ok
test src/str/basics.rs - str::basics::contains (line 191) ... ok
test src/str/basics.rs - str::basics::is_empty (line 35) ... ok
test src/str/basics.rs - str::basics::ends_with (line 110) ... ok
test src/str/basics.rs - str::basics::starts_with (line 70) ... ok
test src/str/difference.rs - str::difference::DifferencePredicate::distance (line 74) ... ok
test src/str/difference.rs - str::difference::DifferencePredicate::split (line 54) ... ok
test src/str/difference.rs - str::difference::diff (line 108) ... ok
test src/str/difference.rs - str::difference::similar (line 131) ... ok
test src/str/regex.rs - str::regex::RegexPredicate::count (line 31) ... ok
test src/str/regex.rs - str::regex::is_match (line 80) ... ok

failures:

failures:
    src/str/adapters.rs - str::adapters::PredicateSliceExt::from_file_path (line 169)

test result: FAILED. 41 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

We verified that it works on a nightly from 20-04-2018.

Meta

rustc --version --verbose:
rustc 1.28.0-nightly (5bf68db 2018-05-28)
binary: rustc
commit-hash: 5bf68db
commit-date: 2018-05-28
host: x86_64-unknown-linux-gnu
release: 1.28.0-nightly
LLVM version: 6.0

[GuillaumeGomez]

Do you know where it could come from @QuietMisdreavus? I don't remember us updating this part of rustdoc since a few months.

[QuietMisdreavus]

Bisecting:

/// ```
/// panic!("oh no");
/// ```
pub struct SomeStruct;

$ rustdoc +nightly-2018-05-13 --test j.rs

running 1 test
test j.rs - SomeStruct (line 1) ... FAILED

failures:

---- j.rs - SomeStruct (line 1) stdout ----
        thread 'j.rs - SomeStruct (line 1)' panicked at 'test executable failed:

thread 'main' panicked at 'oh no', j.rs:3:1
note: Run with `RUST_BACKTRACE=1` for a backtrace.

', librustdoc/test.rs:356:17
note: Run with `RUST_BACKTRACE=1` for a backtrace.


failures:
    j.rs - SomeStruct (line 1)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

$ rustdoc +nightly-2018-05-14 --test j.rs

running 1 test
test j.rs - SomeStruct (line 1) ... FAILED

failures:

failures:
    j.rs - SomeStruct (line 1)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

The only change in src/librustdoc/test.rs in that span was #50235, which touched way more than just rustdoc. cc @Zoxc

[tko]

@QuietMisdreavus looks to me there's a io::set_panic call that should probably be paired with another call to return its original value.

rust/src/librustdoc/test.rs

Lines 267 to 268 in 61ba018

	let old = io::set_panic(Some(box Sink(data.clone())));
	let _bomb = Bomb(data.clone(), old.unwrap_or(box io::stdout()));

[pietroalbini]

Ping @QuietMisdreavus, is there any update on this?

[QuietMisdreavus]

I walked PR-by-PR between those nightlies and found that it was in fact #50235 that caused this. However, that PR doesn't seem to have changed anything about panic handling, or the way things are printed? @rust-lang/compiler I'm a bit lost for what's happening here.

[Mark-Simulacrum]

I think set_panic needs to happen inside each thread and it looks like we might be compiling/running tests inside different threads now in some fashion? I seem to recall such similar behavior last time I looked at a similar issue where the panics during std's tests are printed despite normally being swallowed...

[michaelwoerister]

cc @Zoxc, this is probably not setting some thread-local state?

[Zoxc]

I think the _bomb variable here is the problem. Previously its destructor was ran after the running the test, while now it runs after compilation.