CI: We should be able to complete a build in 3 hours even without Docker image cache

[kennytm]

Our CI builders (except macOS and Windows) use Docker, and we'll cache the Docker repository on Travis. Thanks to the cache, normally the docker build command only takes a few seconds to complete. However, when the cache is invalidated for whatever reason, the Docker image will need to be actually built, and this may take a very long time.

Recently this happened with #49246 — the Docker image cache of dist-x86_64-linux alt became stale and thus needs to be built from scratch. One of the step involves compiling GCC. The whole docker build command thus takes over 40 minutes. Worse, the alt builders have assertions enabled, and thus all stage1+ rustc invocations are slower than their normal counterpart. Together, it is impossible to complete within 3 hours. Travis will not update the cache unless the build is successful. Therefore, I need to exclude RLS, Rustfmt and Clippy from the distribution, to ensure the job is passing.

I don't think we should entirely rely on Travis's cache for speed. Ideally, the docker build command should at most spend 10 minutes, assuming good network speed (~2 MB/s on Travis) and reasonable CPU performance (~2.3 GHz × 4 CPUs on Travis).

In the dist-x86_64-linux alt case, if we host the precompiled GCC 4.8.5 for Centos 5, we could have trimmed 32 minutes out of the Docker build time, which allows us to complete the build without removing anything.

[alexcrichton]

I think the worst offenders here are definitely the dist images where we're commonly building entire gcc toolchains for older compat and whatnot. One of the major pros, I think, is that we can test out changes to toolchains before they land. That is, if you update the toolchain in the CentOS image (for whatever reason) it'll get automatically rebuilt and the PR is rejected if it causes a failure. A downside with precompiled binaries is that we typically don't know if they work, so we'd have to run the precompiled build multiple times.

That being said I think there's a lot we can do to improve this of course! I'd love to have incremental caching of docker images like we have with sccache. With sccache if you send a PR to bors and it fails to build on one platform then all the other platforms are still cached. With Travis's cache then we only cache if the build entirely succeeds, which is tight for things like the docker image updates.

Overall it's basically impossible I think for us to get uncached builds to be under 3 hours, mainly because of dist builders (which require building gcc toolchains) and three (!) versions of LLVM. In that sense I see the main takeaway here is that we should improve the caching strategy with docker layers on Travis, which I've always wanted to do for sure!

[alexcrichton]

Another idea is that this may be a perfect application for Travis's "stages". If we figure out an easy way to share the docker layers between images (aka fast network transfers) then we could split up the docker build stage from the actual rustc build stage. I think that'd far extend our timeout and we'd always comfortably fit within the final time limit.

[alexcrichton]

Er scratch the idea of travis stages, they wouldn't work here. It looks like Travis stages only support parallelism inside one stage, and othewise stages are sequential. That's not quite what we want here...