Reproducible build run-make tests fails if `-C debuginfo=2` (on Linux this time)

[pnkfelix]

This might be a cousin of #66568, which was thought to be Mac-specific.

On my Linux box, with a config.toml that has debug=true and debuginfo-level=2, I do:

 x.py check && \
 x.py test  --stage 1 src/tools/tidy &&  \
 x.py build    --stage 1   library/std  && \
 x.py test    --stage 1    \
     src/test/{rustdoc-js,ui,run-pass-valgrind,mir-opt,codegen,codegen-units,rustdoc,rustdoc-ui,pretty,run-make,run-make-fulldeps,assembly} \
     library/core \
     library/std

After the whole thing runs, it ends up failing in the last step with:

failures:

---- [run-make] run-make-fulldeps/reproducible-build-2 stdout ----

error: make failed
status: exit status: 2
command: "make"
stdout:
------------------------------------------
rm -rf /media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build-2/reproducible-build-2 && mkdir /media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/t\
est/run-make-fulldeps/reproducible-build-2/reproducible-build-2
LD_LIBRARY_PATH="/media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build-2/reproducible-build-2:/media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/\
stage1/lib:/media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/stage0/lib\
" '/media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/stage1/bin/rustc' --out-dir /media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build-2/reprodu\
cible-build-2 -L /media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build-2/reproducible-build-2  reproducible-build-aux.rs
LD_LIBRARY_PATH="/media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build-2/reproducible-build-2:/media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/\
stage1/lib:/media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/stage0/lib\
" '/media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/stage1/bin/rustc' --out-dir /media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build-2/reprodu\
cible-build-2 -L /media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build-2/reproducible-build-2  reproducible-build.rs -C lto=fat
cp /media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build-2/reproducible-build-2/reproducible-build /media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux\
-gnu/test/run-make-fulldeps/reproducible-build-2/reproducible-build-2/reproducible-build-a
LD_LIBRARY_PATH="/media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build-2/reproducible-build-2:/media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/\
stage1/lib:/media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/stage0/lib\
" '/media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/stage1/bin/rustc' --out-dir /media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build-2/reprodu\
cible-build-2 -L /media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build-2/reproducible-build-2  reproducible-build.rs -C lto=fat
cmp "/media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build-2/reproducible-build-2/reproducible-build-a" "/media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown\
-linux-gnu/test/run-make-fulldeps/reproducible-build-2/reproducible-build-2/reproducible-build" || exit 1
/media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/reproducible-build-2/reproducible-build-2/reproducible-build-a /media/pnkfelix/Rust/rust.git/objdir-dbgopt/build/x86_64-unknown-linux-\
gnu/test/run-make-fulldeps/reproducible-build-2/reproducible-build-2/reproducible-build differ: byte 41, line 1

------------------------------------------
stderr:
------------------------------------------
make: *** [Makefile:17: fat_lto] Error 1

------------------------------------------



failures:
    [run-make] run-make-fulldeps/reproducible-build-2

test result: FAILED. 207 passed; 1 failed; 19 ignored; 0 measured; 0 filtered out; finished in 19.77s

I'm filing this issue to track investigation of whether this has the same root cause as #47086 and #66568 (except on Linux now), or if it is something else.

---

• tests/run-make/reproducible-build
• tests/run-make/reproducible-build-2

[pnkfelix]

Seems to occur with just debug-assertions = true in the config.toml

Update: No, I overlooked something else in my own config.toml. Narrowing down now.

[pnkfelix]

OMG I think it needs debuginfo-level = 2 to be observed...

[infinity0]

Same issue on Debian, on all architectures e.g. amd64 ctrl-f for "reproducible-build-2". We set debuginfo-level = 2.

[infinity0]

Might be a consequence of moving to LLVM 13 as the same test passed fine on Debian with 1.55 + LLVM 12, and LLVM is typically responsible for debuginfo. I will try to grab a more detailed diff of the artifacts later.

[infinity0]

Here is detailed list of the differences: https://people.debian.org/~infinity0/rust/reproducible-build-2/diff.txt

Looks like it's ultimately caused by non-deterministic ordering some table of strings, which causes location differences elsewhere.

[jieyouxu]

This is not specific to reproducible-build-2, reproducible-build also observes this failure on Linux if the cases that exercise binary crates + -C debuginfo=2 was not commented out (it also fails on Windows, but binaries already have reproducibility problems without -C debuginfo=2 but with path remapping #88982).