run-make-fulldeps/pgo-branch-weights fails on AArch64 Linux

[pietroalbini]

The most recent builds on AArch64 Linux are failing on this test:

2020-10-22T08:53:26.0713848Z ---- [run-make] run-make-fulldeps/pgo-branch-weights stdout ----
2020-10-22T08:53:26.0723060Z 
2020-10-22T08:53:26.0727694Z error: make failed
2020-10-22T08:53:26.0732974Z status: exit code: 2
2020-10-22T08:53:26.0738071Z command: "make"
2020-10-22T08:53:26.0742831Z stdout:
2020-10-22T08:53:26.0750881Z ------------------------------------------
2020-10-22T08:53:26.0763884Z # We don't compile `opaque` with either optimizations or instrumentation.
2020-10-22T08:53:26.0783737Z LD_LIBRARY_PATH="/checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights:/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/lib:/checkout/obj/build/aarch64-unknown-linux-gnu/stage0-bootstrap-tools/aarch64-unknown-linux-gnu/release/deps:/checkout/obj/build/aarch64-unknown-linux-gnu/stage0/lib" '/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc' --out-dir /checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights -L /checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights   opaque.rs || exit 1
2020-10-22T08:53:26.0791260Z # Compile the test program with instrumentation
2020-10-22T08:53:26.0794012Z mkdir -p "/checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights/prof_data_dir" || exit 1
2020-10-22T08:53:26.0803441Z LD_LIBRARY_PATH="/checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights:/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/lib:/checkout/obj/build/aarch64-unknown-linux-gnu/stage0-bootstrap-tools/aarch64-unknown-linux-gnu/release/deps:/checkout/obj/build/aarch64-unknown-linux-gnu/stage0/lib" '/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc' --out-dir /checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights -L /checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights   interesting.rs \
2020-10-22T08:53:26.0813242Z 	-Cprofile-generate="/checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights/prof_data_dir" -O -Ccodegen-units=1 || exit 1
2020-10-22T08:53:26.0824507Z LD_LIBRARY_PATH="/checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights:/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/lib:/checkout/obj/build/aarch64-unknown-linux-gnu/stage0-bootstrap-tools/aarch64-unknown-linux-gnu/release/deps:/checkout/obj/build/aarch64-unknown-linux-gnu/stage0/lib" '/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc' --out-dir /checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights -L /checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights   main.rs -Cprofile-generate="/checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights/prof_data_dir" -O || exit 1
2020-10-22T08:53:26.0833472Z # The argument below generates to the expected branch weights
2020-10-22T08:53:26.0841671Z LD_LIBRARY_PATH="/checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights:/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/lib/rustlib/aarch64-unknown-linux-gnu/lib:/checkout/obj/build/aarch64-unknown-linux-gnu/stage0-bootstrap-tools/aarch64-unknown-linux-gnu/release/deps:/checkout/obj/build/aarch64-unknown-linux-gnu/stage0/lib" /checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights/main aaaaaaaaaaaa2bbbbbbbbbbbb2bbbbbbbbbbbbbbbbcc || exit 1
2020-10-22T08:53:26.0849409Z "/checkout/obj/build/aarch64-unknown-linux-gnu/llvm/build/bin/llvm-profdata" merge \
2020-10-22T08:53:26.0852410Z 	-o "/checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights/prof_data_dir/merged.profdata" \
2020-10-22T08:53:26.0855865Z 	"/checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights/prof_data_dir" || exit 1
2020-10-22T08:53:26.0864853Z LD_LIBRARY_PATH="/checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights:/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/lib:/checkout/obj/build/aarch64-unknown-linux-gnu/stage0-bootstrap-tools/aarch64-unknown-linux-gnu/release/deps:/checkout/obj/build/aarch64-unknown-linux-gnu/stage0/lib" '/checkout/obj/build/aarch64-unknown-linux-gnu/stage2/bin/rustc' --out-dir /checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights -L /checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights   interesting.rs \
2020-10-22T08:53:26.0874373Z 	-Cprofile-use="/checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights/prof_data_dir/merged.profdata" -O \
2020-10-22T08:53:26.0876942Z 	-Ccodegen-units=1 --emit=llvm-ir || exit 1
2020-10-22T08:53:26.0880673Z cat "/checkout/obj/build/aarch64-unknown-linux-gnu/test/run-make-fulldeps/pgo-branch-weights/pgo-branch-weights/interesting.ll" | "/checkout/obj/build/aarch64-unknown-linux-gnu/llvm/build/bin/FileCheck" filecheck-patterns.txt
2020-10-22T08:53:26.0883188Z 
2020-10-22T08:53:26.0884121Z ------------------------------------------
2020-10-22T08:53:26.0884722Z stderr:
2020-10-22T08:53:26.0885663Z ------------------------------------------
2020-10-22T08:53:26.0887152Z filecheck-patterns.txt:5:8: error: CHECK: expected string not found in input
2020-10-22T08:53:26.0889855Z CHECK: define void @function_called_twice(i32 %c) {{.*}} !prof [[function_called_twice_id:![0-9]+]] {
2020-10-22T08:53:26.0890870Z        ^
2020-10-22T08:53:26.0891548Z <stdin>:1:1: note: scanning from here
2020-10-22T08:53:26.0893037Z ; ModuleID = 'interesting.3a1fbbbh-cgu.0'
2020-10-22T08:53:26.0893984Z ^
2020-10-22T08:53:26.0894732Z <stdin>:7:1: note: possible intended match here
2020-10-22T08:53:26.0895788Z define void @function_called_twice(i32 %c) unnamed_addr #0 {
2020-10-22T08:53:26.0896578Z ^
2020-10-22T08:53:26.0896922Z 
2020-10-22T08:53:26.0897496Z Input file: <stdin>
2020-10-22T08:53:26.0898859Z Check file: filecheck-patterns.txt
2020-10-22T08:53:26.0899576Z 
2020-10-22T08:53:26.0900782Z -dump-input=help explains the following input dump.
2020-10-22T08:53:26.0901455Z 
2020-10-22T08:53:26.0902005Z Input was:
2020-10-22T08:53:26.0902571Z <<<<<<
2020-10-22T08:53:26.0903979Z            1: ; ModuleID = 'interesting.3a1fbbbh-cgu.0'
2020-10-22T08:53:26.0905561Z check:5'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
2020-10-22T08:53:26.0908184Z            2: source_filename = "interesting.3a1fbbbh-cgu.0"
2020-10-22T08:53:26.0909684Z check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-10-22T08:53:26.0911280Z            3: target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
2020-10-22T08:53:26.0912831Z check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-10-22T08:53:26.0914249Z            4: target triple = "aarch64-unknown-linux-gnu"
2020-10-22T08:53:26.0915583Z check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-10-22T08:53:26.0916181Z            5: 
2020-10-22T08:53:26.0917010Z check:5'0     ~
2020-10-22T08:53:26.0918142Z            6: ; Function Attrs: noinline nonlazybind uwtable
2020-10-22T08:53:26.0919605Z check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-10-22T08:53:26.0920542Z            7: define void @function_called_twice(i32 %c) unnamed_addr #0 {
2020-10-22T08:53:26.0921934Z check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-10-22T08:53:26.0923158Z check:5'1     ?                                                            possible intended match
2020-10-22T08:53:26.0923948Z            8: start:
2020-10-22T08:53:26.0924831Z check:5'0     ~~~~~~
2020-10-22T08:53:26.0925481Z            9:  %0 = icmp eq i32 %c, 50
2020-10-22T08:53:26.0926480Z check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~
2020-10-22T08:53:26.0927169Z           10:  br i1 %0, label %bb2, label %bb1
2020-10-22T08:53:26.0928608Z check:5'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-10-22T08:53:26.0929312Z           11: 
2020-10-22T08:53:26.0930503Z check:5'0     ~
2020-10-22T08:53:26.0931285Z           12: bb1: ; preds = %start
2020-10-22T08:53:26.0932694Z check:5'0     ~~~~~~~~~~~~~~~~~~~~~
2020-10-22T08:53:26.0933375Z            .
2020-10-22T08:53:26.0934086Z            .
2020-10-22T08:53:26.0934834Z            .
2020-10-22T08:53:26.0935442Z >>>>>>
2020-10-22T08:53:26.0936135Z make: *** [Makefile:42: all] Error 1
2020-10-22T08:53:26.0936642Z 
2020-10-22T08:53:26.0937694Z ------------------------------------------
2020-10-22T08:53:26.0938123Z 
2020-10-22T08:53:26.0938461Z 
2020-10-22T08:53:26.0938808Z 
2020-10-22T08:53:26.0939349Z failures:
2020-10-22T08:53:26.0940684Z     [run-make] run-make-fulldeps/pgo-branch-weights
2020-10-22T08:53:26.0941493Z 
2020-10-22T08:53:26.0942824Z test result: FAILED. 211 passed; 1 failed; 7 ignored; 0 measured; 0 filtered out

[pietroalbini]

@rustbot ping arm

[rustbot]

Hey ARM Group! This bug has been identified as a good "ARM candidate".
In case it's useful, here are some instructions for tackling these sorts of
bugs. Maybe take a look?
Thanks! <3

cc @JamieCunliffe @joaopaulocarreiro @raw-bin @Stammark @vigoux

[JohnTitor]

Looks like my rollup (#78212) has the cause.

[raw-bin]

@JohnTitor: Thanks for the tip! We're looking into this.

[JamieCunliffe]

Based on that I reverted #77554 (git revert 813066c4429e7582b08dbf3af2c12a5f2e1b2a16 -m 1) and the test then passed.

[pietroalbini]

This seems to be fixed now, CI is passing again.

[Mark-Simulacrum]

Reopening this issue as it's hitting us (again) on the beta promotion PR, #86413.

I've tried various things to debug but so far haven't really arrived at any conclusions. It doesn't seem immediately related to anything in the PR itself, which should in theory have minimal effect on runtime behavior here.

@rustbot ping arm -- since this is only failing on the aarch64 builder, I am currently presuming some relationship, but it's not clear. The test does not fail on the only aarch64 machine I currently readily have access to when I tried to run it indirectly (i.e., copying the input files and simulating the make file via some local manual invocations)>

[rustbot]

Error: Parsing ping command in comment failed: ...'t ping arm' | error: expected end of command at >| ' -- since '...

Please let @rust-lang/release know if you're having trouble with this bot.

[Mark-Simulacrum]

@rustbot ping arm