compiletest failures are close to unreadable because of JSON

[brson]

Here's one I hit recently:

---- [run-pass] run-pass/hello.rs stdout ----                                                                                                                                             [124/9070]

error: compilation failed!
status: exit code: 101
command: /mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/bin/rustc /mnt2/dev/rust/src/test/run-pass/hello.rs -L /mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass --target=asmjs-unkn
own-emscripten --error-format json -L /mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/hello.stage2-asmjs-unknown-emscripten.run-pass.libaux -C prefer-dynamic -o /mnt2/dev/rust/build/x8
6_64-unknown-linux-gnu/test/run-pass/hello.stage2-asmjs-unknown-emscripten.js -Crpath -O -Lnative=/mnt2/dev/rust/build/asmjs-unknown-emscripten/rust-test-helpers
stdout:
------------------------------------------

------------------------------------------
stderr:
------------------------------------------
{"message":"linking with `emcc` failed: exit code: 1","code":null,"level":"error","spans":[],"children":[{"message":"\"emcc\" \"-L\" \"/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rust
lib/asmjs-unknown-emscripten/lib\" \"/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/hello.0.o\" \"-o\" \"/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass/hello.stage2-asmjs
-unknown-emscripten.js\" \"-Wl,--gc-sections\" \"-Wl,-O1\" \"-nodefaultlibs\" \"-L\" \"/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/test/run-pass\" \"-L\" \"/mnt2/dev/rust/build/x86_64-unknown-li
nux-gnu/test/run-pass/hello.stage2-asmjs-unknown-emscripten.run-pass.libaux\" \"-L\" \"/mnt2/dev/rust/build/asmjs-unknown-emscripten/rust-test-helpers\" \"-L\" \"/mnt2/dev/rust/build/x86_64-unknow
n-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripten/lib\" \"-Wl,-Bstatic\" \"-Wl,-Bdynamic\" \"/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripten/lib/lib
std-c5ae712d21ff1b88.rlib\" \"/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripten/lib/libpanic_unwind-0aed89c908dbae00.rlib\" \"/mnt2/dev/rust/build/x86_64-un
known-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripten/lib/libunwind-4406c7bac693a305.rlib\" \"/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripten/lib/li
brand-4bf39e147f01f701.rlib\" \"/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripten/lib/libcollections-94042b4fb30df9b8.rlib\" \"/mnt2/dev/rust/build/x86_64-u
nknown-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripten/lib/librustc_unicode-e09cf12a6737fbe8.rlib\" \"/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripte
n/lib/liballoc-e2faee10ccc53a4e.rlib\" \"/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripten/lib/liballoc_system-57a84a70354893c6.rlib\" \"/mnt2/dev/rust/buil
d/x86_64-unknown-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripten/lib/liblibc-5f7c0b0b6e65da09.rlib\" \"/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscript
en/lib/libcore-380af50841193d3f.rlib\" \"/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripten/lib/libcompiler_builtins-3720833c50b88cf0.rlib\" \"-l\" \"c\" \"-
s\" \"ERROR_ON_UNDEFINED_SYMBOLS=1\"","code":null,"level":"note","spans":[],"children":[],"rendered":null},{"message":"/mnt2/dev/emsdk_portable/clang/fastcomp/build_incoming_64/bin/llvm-nm: /mnt2/
dev/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripten/lib/libstd-c5ae712d21ff1b88.rlib(rust.metadata.bin) The end of the file was unexpectedly encountered\n\n/mnt2/de
v/emsdk_portable/clang/fastcomp/build_incoming_64/bin/llvm-nm: /mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripten/lib/libpanic_unwind-0aed89c908dbae00.rlib(r
ust.metadata.bin) The end of the file was unexpectedly encountered\n\n/mnt2/dev/emsdk_portable/clang/fastcomp/build_incoming_64/bin/llvm-nm: /mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/li
b/rustlib/asmjs-unknown-emscripten/lib/libunwind-4406c7bac693a305.rlib(rust.metadata.bin) The end of the file was unexpectedly encountered\n\n/mnt2/dev/emsdk_portable/clang/fastcomp/build_incoming
_64/bin/llvm-nm: /mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripten/lib/librand-4bf39e147f01f701.rlib(rust.metadata.bin) The end of the file was unexpectedly
 encountered\n\n/mnt2/dev/emsdk_portable/clang/fastcomp/build_incoming_64/bin/llvm-nm: /mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripten/lib/libcollections-
94042b4fb30df9b8.rlib(rust.metadata.bin) The end of the file was unexpectedly encountered\n\n/mnt2/dev/emsdk_portable/clang/fastcomp/build_incoming_64/bin/llvm-nm: /mnt2/dev/rust/build/x86_64-unkn
own-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripten/lib/librustc_unicode-e09cf12a6737fbe8.rlib(rust.metadata.bin) The end of the file was unexpectedly encountered\n\n/mnt2/dev/emsdk_portable
/clang/fastcomp/build_incoming_64/bin/llvm-nm: /mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripten/lib/liballoc-e2faee10ccc53a4e.rlib(rust.metadata.bin) The e
nd of the file was unexpectedly encountered\n\n/mnt2/dev/emsdk_portable/clang/fastcomp/build_incoming_64/bin/llvm-nm: /mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/asmjs-unknown
-emscripten/lib/liballoc_system-57a84a70354893c6.rlib(rust.metadata.bin) The end of the file was unexpectedly encountered\n\n/mnt2/dev/emsdk_portable/clang/fastcomp/build_incoming_64/bin/llvm-nm:
/mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripten/lib/liblibc-5f7c0b0b6e65da09.rlib(rust.metadata.bin) The end of the file was unexpectedly encountered\n\n/
mnt2/dev/emsdk_portable/clang/fastcomp/build_incoming_64/bin/llvm-nm: /mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/asmjs-unknown-emscripten/lib/libcore-380af50841193d3f.rlib(ru
st.metadata.bin) The end of the file was unexpectedly encountered\n\n/mnt2/dev/emsdk_portable/clang/fastcomp/build_incoming_64/bin/llvm-nm: /mnt2/dev/rust/build/x86_64-unknown-linux-gnu/stage2/lib
/rustlib/asmjs-unknown-emscripten/lib/libcompiler_builtins-3720833c50b88cf0.rlib(rust.metadata.bin) The end of the file was unexpectedly encountered\n\nerror: unresolved symbol: _Unwind_GetRegionS
tart\nerror: unresolved symbol: _Unwind_SetGR\nerror: unresolved symbol: _Unwind_GetLanguageSpecificData\nerror: unresolved symbol: _Unwind_SetIP\nerror: unresolved symbol: _Unwind_GetTextRelBase\
nerror: unresolved symbol: _Unwind_GetDataRelBase\nAborting compilation due to previous errors | undefined\nTraceback (most recent call last):\n  File \"/mnt2/dev/emsdk_portable/emscripten/incomin
g/emcc\", line 13, in <module>\n    emcc.run()\n  File \"/mnt2/dev/emsdk_portable/emscripten/incoming/emcc.py\", line 1510, in run\n    final = shared.Building.emscripten(final, append_ext=False,
extra_args=extra_args)\n  File \"/mnt2/dev/emsdk_portable/emscripten/incoming/tools/shared.py\", line 1707, in emscripten\n    call_emscripten(cmdline)\n  File \"/mnt2/dev/emsdk_portable/emscripte
n/incoming/emscripten.py\", line 1756, in _main\n    temp_files.run_and_clean(lambda: main(\n  File \"/mnt2/dev/emsdk_portable/emscripten/incoming/tools/tempfiles.py\", line 78, in run_and_clean\n
    return func()\n  File \"/mnt2/dev/emsdk_portable/emscripten/incoming/emscripten.py\", line 1761, in <lambda>\n    DEBUG=DEBUG,\n  File \"/mnt2/dev/emsdk_portable/emscripten/incoming/emscripten
.py\", line 1662, in main\n    temp_files=temp_files, DEBUG=DEBUG)\n  File \"/mnt2/dev/emsdk_portable/emscripten/incoming/emscripten.py\", line 91, in emscript\n    glue, forwarded_data = compiler
_glue(metadata, settings, libraries, compiler_engine, temp_files, DEBUG)\n  File \"/mnt2/dev/emsdk_portable/emscripten/incoming/emscripten.py\", line 286, in compiler_glue\n    cwd=path_from_root(
'src'), error_limit=300)\n  File \"/mnt2/dev/emsdk_portable/emscripten/incoming/tools/jsrun.py\", line 76, in run_js\n    raise Exception('Expected the command ' + str(command) + ' to finish with
return code ' + str(assert_returncode) + ', but it returned with code ' + str(proc.returncode) + ' instead! Output: ' + str(ret)[:error_limit])\nException: Expected the command ['/mnt2/dev/emsdk_p
ortable/node/4.1.1_64bit/bin/node', '/mnt2/dev/emsdk_portable/emscripten/incoming/src/compiler.js', '/tmp/tmpnGlx_B.txt', '/mnt2/dev/emsdk_portable/emscripten/incoming/src/library_pthread_stub.js'
] to finish with return code 0, but it returned with code 1 instead! Output: // The Module object: Our interface to the outside world. We import\n// and export values on it, and do the work to get
 that through\n// closure compiler if necessary. There are various ways Module can be used:\n// 1. Not defined. We create it here\n// 2. A function parameter, function(Module) { ..gener\n","code":
null,"level":"note","spans":[],"children":[],"rendered":null}],"rendered":null}
{"message":"aborting due to previous error","code":null,"level":"error","spans":[],"children":[],"rendered":null}

------------------------------------------

thread '[run-pass] run-pass/hello.rs' panicked at 'explicit panic', src/tools/compiletest/src/runtest.rs:2359
note: Run with `RUST_BACKTRACE=1` for a backtrace.

compiletest uses the --error-format=json switch to capture more precise error information then dumps that out for humans to read.

Instead we might either: have the compiler output the json to a side file; reformat the JSON into something that makes more sense.

[brson]

@nikomatsakis suggests that the compile-fail output is better and json is only used for that case, so compiletest could be modified to only use json mode there.

[brson]

I've started a patch.

[brson]

Simply disabling json errors for rpass isn't sufficient because there are rpass errors that use the new precise error checking to check warnings.

[nikomatsakis]

@brson example? Perhaps those tests should be made into ui tests? I forget if those work for run-pass tests or not.

In general, as discussed on this thread, I'd like to move over to ui testing more and more. One of the things I had hoped to do was to generalize the concept of "run-pass" vs "compile-fail" test so that we just had "tests", where you could specify in the test how it was expected to behave (compile successfully, compile with errors, etc) and then use a reference file to check the compiler's output. I don't claim to have this vision 100% worked out yet though.

[brson]

I don't think I'll get to fixing this after all because of the aforementioned issue.

[brson]

@nikomatsakis here are the rpass failures that check warnings:

    [run-pass] run-pass/deprecated-macro_escape-inner.rs
    [run-pass] run-pass/deprecated-macro_escape.rs
    [run-pass] run-pass/deriving-meta-empty-trait-list.rs
    [run-pass] run-pass/enum-size-variance.rs
    [run-pass] run-pass/issue-19100.rs

[alexcrichton]

@brson do you have an update on this, just hit this again with run-pass-fulldeps tests unfortunately :(

@nikomatsakis I think the problem here is that the tests aren't UI tests, just standard run-pass tests. Everything is json errors now which makes it harder to diagnose failures.

[nikomatsakis]

@alexcrichton my point is that I think we could switch run-pass to not use JSON. If the sticking point is those 5 tests, we could either fix them somehow, or convert those 5 into ui tests instead.

[brson]

@alexcrichton No, I'm not working on this.

[alexcrichton]

@nikomatsakis agreed! (that we can switch)