Automate management of broken tools

[nrc]

Rather than using the tools manifest for opting-in to breaking a tool.

The rough plan: on master (but not beta or stable) the CI would ignore failing to build or test some tools (RLS, Clippy, etc). However, some bot would note when a tool started to fail and email interested parties (and/or display the info in a dashboard).

This should lower the friction for contributors working in the Rust repo when they make tool-breaking changes. However, we might miss some 'true positive' failure in the tools (e.g., #45820).

[kennytm]

We could do it like this on the Travis CI side:

1. Change the purpose of the cargotest job and aux job.

   • The old cargotest job becomes the new aux job. It will run:
     • cargotest
     • pretty tests
   • The old aux job becomes the tools job. It will run test against external tools:
     • cargo, rls, rustfmt, miri, clippy

2. Don't run the tools job in auto branch. (Promote something to tier-2 to take its place)

   • Alternatively, still run the job but mark it as allow_failure. That's rather pointless though.

3. Run the tools job whenever the branch is master/beta/stable and not a PR.

   • We could also run this in a PR, so reviewer can inspect the ❌ and determine whether it is legit or spurious, to reduce the chance a "true positive" is missed. This will double the resource spent on testing PRs, and won't reflect the actual behavior when merged to master.

4. Monitor the Travis badge.

Rough sketch of the change:

diff --git a/.travis.yml b/.travis.yml
index db34f14044..a28954761b 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -161,16 +161,16 @@ matrix:
       if: branch = auto
     - env: IMAGE=i686-gnu-nopt
       if: branch = auto
-    # - env: IMAGE=wasm32 issue 42646
-    #   if: branch = auto
+    - env: IMAGE=wasm32
+      if: branch = auto
     - env: IMAGE=x86_64-gnu
       if: branch = auto
     - env: IMAGE=x86_64-gnu-full-bootstrap
       if: branch = auto
+    - env: IMAGE=x86_64-gnu-tools
+      if: branch IN (master, beta, stable) # (<- uncomment to disable PR test) AND type != pull_request
     - env: IMAGE=x86_64-gnu-aux
       if: branch = auto
-    - env: IMAGE=x86_64-gnu-cargotest
-      if: branch = auto
     - env: IMAGE=x86_64-gnu-debug
       if: branch = auto
     - env: IMAGE=x86_64-gnu-nopt

Not sure what will be the corresponding change on the AppVeyor side.

[kennytm]

Ideas after discussion with @nrc,

5. We will create a repository to store the current test status. The Travis+AppVeyor job will push a commit whenever the test result changes. The dev-tools team will create a dashboard to read from this repository.

6. The commit message will @-mention any party responsible for the tool.

[nrc]

We discussed this at the dev-tools meeting today. We mostly discussed the implementation details. One interesting issue that came up: we want to avoid notification except on changing from testing to not testing. That requires some state in the 'bot' doing the monitoring. One way to avoid security concerns is to use a GitHub repo (I've done this for rustaceans.org and findwork). We might then be able to rely on GitHub notifications for doing the email notification when a tool breaks.

[kennytm]

Tracker.

Changes to the rust-lang/rust CI system

• Ensure dist jobs can still function correctly even when a tool is broken (ignoring toolstate.toml) ([auto-toolstate][1/8] Always ignore build failure of failable tools (rls, rustfmt, clippy) #46102)

• Perform the cargotest → aux and aux → tools rotation as outlined in point 1 in Automate management of broken tools #45861 (comment). ([auto-toolstate][2+3/8] Move external tools tests into its own job with --no-fail-fast #46393)

• Ensure the tools jobs is not fail-fast; collect test result into a local machine-readable file (Probably just a CSV file so that it can be append-only). ([auto-toolstate][2+3/8] Move external tools tests into its own job with --no-fail-fast #46393)

• Create a GitHub repository to store the test result, and a secret bot token to push to this repository.

• Let tools job upload the test result:

  • If the current channel is not nightly (e.g. beta or stable), exit the CI job so bors can prevent it merging.
  • If the current channel is nightly, append the test result and push to the toolstate-records repository. Since both AppVeyor and Travis will push to the same repository, the compare-and-push action must be --ff-only and needs several retries with jitter.

• Create a new lightweight CI job on master that promotes a result from the toolstate-records repo. If the result differ, create a commit with @-mentions.

• Update rustbuild to remove traces of toolstate.toml.

• Update documentation.

(Note: we need to disable tools jobs from auto before enabling @-mentions to prevent spurious mentions due to failing builds.)

Other changes

• Build a dashboard using the content of the new repository?