azure: Get build times on par with Travis

[alexcrichton]

Currently the build times for Azure Pipelines are through the roof compared to Travis, taking up to an hour longer on relevant builds. The original PR has some measurements.

We're going to have to rework quite a lot I think to get the current 2-core within our time limits. We're shooting to have everything in 2h on a good day, and up to 3h if necessary to do things like rebuild caches and such.

[alexcrichton]

Current analysis of build time discrepancies

[alexcrichton]

Ok to expand on the comment linked above, I've run azure again but this time with step timings enabled for the no assertions build, yielding this chart:

https://gist.github.com/alexcrichton/46531c79f85fc213c9cd9f7417c3b1ec

Here are some comparisons of build times between Travis/Appveyor to Azure. A + sign means azure is slower for a step, and a - means azure is faster.

Build time discrepancies of dist-i686-msvc:

+  10.162 (+ 33.01% /   30.782 =>   40.944) Lld { target: "i686-pc-windows-msvc" }
-  12.264 (- 12.65% /   96.941 =>   84.677) Std { compiler: Compiler { stage: 2, host: "i686-pc-windows-msvc" }, target: "i686-pc-windows-msvc" }
+  13.310 (+ 12.14% /  109.674 =>  122.984) Docs { host: "i686-pc-windows-msvc" }
-  16.205 (- 11.88% /  136.388 =>  120.183) Docs { host: "i586-pc-windows-msvc" }
+  20.911 (+ 21.01% /   99.542 =>  120.453) ToolBuild { compiler: Compiler { stage: 0, host: "i686-pc-windows-msvc" }, target: "i686-pc-windows-msvc", tool: "unstable-book-gen", path: "src/tools/unstable-book-gen", mode: ToolBootstrap, is_optional_tool: false, source_type: InTree, extra_features: [] }
+  22.946 (+ 29.75% /   77.117 =>  100.063) ToolBuild { compiler: Compiler { stage: 1, host: "i686-pc-windows-msvc" }, target: "i686-pc-windows-msvc", tool: "miri", path: "src/tools/miri", mode: ToolRustc, is_optional_tool: true, source_type: Submodule, extra_features: [] }
+  22.988 (+ 28.63% /   80.281 =>  103.269) Std { target: "i586-pc-windows-msvc", compiler: Compiler { stage: 2, host: "i686-pc-windows-msvc" } }
+  23.417 (+ 23.11% /  101.307 =>  124.724) Std { stage: 2, target: "i686-pc-windows-msvc" }
-  23.794 (- 21.75% /  109.385 =>   85.591) Std { compiler: Compiler { stage: 1, host: "i686-pc-windows-msvc" }, target: "i686-pc-windows-msvc" }
+  25.255 (+ 24.88% /  101.513 =>  126.768) Std { stage: 2, target: "i586-pc-windows-msvc" }
+  27.378 (+ 42.84% /   63.906 =>   91.284) ToolBuild { compiler: Compiler { stage: 0, host: "i686-pc-windows-msvc" }, target: "i686-pc-windows-msvc", tool: "fabricate", path: "src/tools/rust-installer", mode: ToolBootstrap, is_optional_tool: false, source_type: Submodule, extra_features: [] }
+  31.480 (+ 38.73% /   81.286 =>  112.766) Std { target: "i686-pc-windows-msvc", compiler: Compiler { stage: 1, host: "i686-pc-windows-msvc" } }
-  31.993 (- 24.46% /  130.795 =>   98.802) Rustc { compiler: Compiler { stage: 2, host: "i686-pc-windows-msvc" } }
+  32.202 (+ 36.58% /   88.034 =>  120.236) CodegenBackend { compiler: Compiler { stage: 1, host: "i686-pc-windows-msvc" }, target: "i686-pc-windows-msvc", backend: "emscripten" }
+  41.841 (+ 38.43% /  108.882 =>  150.723) CodegenBackend { compiler: Compiler { stage: 1, host: "i686-pc-windows-msvc" }, target: "i686-pc-windows-msvc", backend: "llvm" }
+  42.923 (+ 47.46% /   90.433 =>  133.356) CodegenBackend { compiler: Compiler { stage: 0, host: "i686-pc-windows-msvc" }, target: "i686-pc-windows-msvc", backend: "emscripten" }
+  52.187 (+ 45.71% /  114.179 =>  166.366) CodegenBackend { compiler: Compiler { stage: 0, host: "i686-pc-windows-msvc" }, target: "i686-pc-windows-msvc", backend: "llvm" }
+  65.974 (+ 45.22% /  145.911 =>  211.885) Rustdoc { compiler: Compiler { stage: 1, host: "i686-pc-windows-msvc" } }
+  90.244 (+ 61.80% /  146.024 =>  236.268) ToolBuild { compiler: Compiler { stage: 2, host: "i686-pc-windows-msvc" }, target: "i686-pc-windows-msvc", tool: "error_index_generator", path: "src/tools/error_index_generator", mode: ToolRustc, is_optional_tool: false, source_type: InTree, extra_features: [] }
+  94.083 (+ 66.21% /  142.091 =>  236.174) Rustdoc { compiler: Compiler { stage: 2, host: "i686-pc-windows-msvc" } }
+  95.547 (+ 41.23% /  231.750 =>  327.297) ToolBuild { compiler: Compiler { stage: 1, host: "i686-pc-windows-msvc" }, target: "i686-pc-windows-msvc", tool: "rustfmt", path: "src/tools/rustfmt", mode: ToolRustc, is_optional_tool: true, source_type: Submodule, extra_features: [] }
+ 121.531 (+ 30.71% /  395.718 =>  517.249) ToolBuild { compiler: Compiler { stage: 0, host: "i686-pc-windows-msvc" }, target: "i686-pc-windows-msvc", tool: "rustbook", path: "src/tools/rustbook", mode: ToolBootstrap, is_optional_tool: false, source_type: InTree, extra_features: [] }
+ 152.714 (+ 55.17% /  276.824 =>  429.538) Llvm { target: "i686-pc-windows-msvc", emscripten: true }
+ 161.283 (+ 27.23% /  592.311 =>  753.594) ToolBuild { compiler: Compiler { stage: 1, host: "i686-pc-windows-msvc" }, target: "i686-pc-windows-msvc", tool: "cargo", path: "src/tools/cargo", mode: ToolRustc, is_optional_tool: false, source_type: Submodule, extra_features: [] }
+ 196.614 (+ 59.18% /  332.226 =>  528.840) ToolBuild { compiler: Compiler { stage: 1, host: "i686-pc-windows-msvc" }, target: "i686-pc-windows-msvc", tool: "clippy-driver", path: "src/tools/clippy", mode: ToolRustc, is_optional_tool: true, source_type: Submodule, extra_features: [] }
+ 244.891 (+ 50.01% /  489.680 =>  734.571) ToolBuild { compiler: Compiler { stage: 1, host: "i686-pc-windows-msvc" }, target: "i686-pc-windows-msvc", tool: "rls", path: "src/tools/rls", mode: ToolRustc, is_optional_tool: true, source_type: Submodule, extra_features: ["clippy"] }
+ 272.407 (+ 58.82% /  463.139 =>  735.546) Llvm { target: "i686-pc-windows-msvc", emscripten: false }
+ 481.533 (+ 36.43% / 1321.943 => 1803.476) Rustc { target: "i686-pc-windows-msvc", compiler: Compiler { stage: 0, host: "i686-pc-windows-msvc" } }
+ 692.815 (+ 53.80% / 1287.733 => 1980.548) Rustc { target: "i686-pc-windows-msvc", compiler: Compiler { stage: 1, host: "i686-pc-windows-msvc" } }

Build time discrepancies of dist-x86_64-apple

+  10.071 (+ 31.61% /   31.861 =>   41.932) Test { target: "i386-apple-ios", compiler: Compiler { stage: 2, host: "x86_64-apple-darwin" } }
+  11.843 (+ 14.50% /   81.682 =>   93.525) Std { target: "aarch64-apple-ios", compiler: Compiler { stage: 2, host: "x86_64-apple-darwin" } }
+  12.079 (+ 37.94% /   31.835 =>   43.914) Test { target: "x86_64-apple-ios", compiler: Compiler { stage: 2, host: "x86_64-apple-darwin" } }
+  13.058 (+ 20.50% /   63.707 =>   76.765) CodegenBackend { compiler: Compiler { stage: 1, host: "x86_64-apple-darwin" }, target: "x86_64-apple-darwin", backend: "emscripten" }
+  13.061 (+ 39.76% /   32.850 =>   45.911) Test { target: "armv7s-apple-ios", compiler: Compiler { stage: 2, host: "x86_64-apple-darwin" } }
-  15.202 (-  8.69% /  174.960 =>  159.758) ToolBuild { compiler: Compiler { stage: 1, host: "x86_64-apple-darwin" }, target: "x86_64-apple-darwin", tool: "rustfmt", path: "src/tools/rustfmt", mode: ToolRustc, is_optional_tool: true, source_type: Submodule, extra_features: [] }
-  16.505 (-  6.71% /  245.924 =>  229.419) ToolBuild { compiler: Compiler { stage: 1, host: "x86_64-apple-darwin" }, target: "x86_64-apple-darwin", tool: "clippy-driver", path: "src/tools/clippy", mode: ToolRustc, is_optional_tool: true, source_type: Submodule, extra_features: [] }
+  16.586 (+ 51.05% /   32.487 =>   49.073) Test { target: "x86_64-apple-darwin", compiler: Compiler { stage: 0, host: "x86_64-apple-darwin" } }
+  17.338 (+ 13.70% /  126.557 =>  143.895) Rustdoc { compiler: Compiler { stage: 1, host: "x86_64-apple-darwin" } }
+  17.872 (+ 17.94% /   99.628 =>  117.500) Docs { host: "armv7s-apple-ios" }
+  20.619 (+ 17.80% /  115.840 =>  136.459) Std { stage: 2, target: "aarch64-apple-ios" }
+  20.842 (+  5.15% /  404.398 =>  425.240) Extended { stage: 2, host: "x86_64-apple-darwin", target: "x86_64-apple-darwin" }
+  20.966 (+ 23.92% /   87.665 =>  108.631) Std { target: "x86_64-apple-darwin", compiler: Compiler { stage: 0, host: "x86_64-apple-darwin" } }
+  22.619 (+ 19.22% /  117.704 =>  140.323) Std { stage: 2, target: "armv7s-apple-ios" }
+  23.810 (+ 28.63% /   83.179 =>  106.989) Docs { host: "aarch64-apple-ios" }
+  24.189 (+ 20.03% /  120.743 =>  144.932) Std { stage: 2, target: "x86_64-apple-ios" }
+  25.506 (+  3.79% /  672.437 =>  697.943) Llvm { target: "x86_64-apple-darwin", emscripten: false }
+  36.363 (+ 31.94% /  113.837 =>  150.200) Std { stage: 2, target: "armv7-apple-ios" }
+  42.906 (+ 14.21% /  302.043 =>  344.949) ToolBuild { compiler: Compiler { stage: 1, host: "x86_64-apple-darwin" }, target: "x86_64-apple-darwin", tool: "rls", path: "src/tools/rls", mode: ToolRustc, is_optional_tool: true, source_type: Submodule, extra_features: ["clippy"] }
+  46.882 (+ 24.69% /  189.871 =>  236.753) Llvm { target: "x86_64-apple-darwin", emscripten: true }
+  55.398 (+ 44.87% /  123.474 =>  178.872) Rustdoc { compiler: Compiler { stage: 2, host: "x86_64-apple-darwin" } }
+  56.766 (+ 42.61% /  133.213 =>  189.979) ToolBuild { compiler: Compiler { stage: 2, host: "x86_64-apple-darwin" }, target: "x86_64-apple-darwin", tool: "error_index_generator", path: "src/tools/error_index_generator", mode: ToolRustc, is_optional_tool: false, source_type: InTree, extra_features: [] }
+  58.042 (+ 47.88% /  121.222 =>  179.264) Std { stage: 2, target: "i386-apple-ios" }
+  74.360 (+124.70% /   59.631 =>  133.991) ToolBuild { compiler: Compiler { stage: 0, host: "x86_64-apple-darwin" }, target: "x86_64-apple-darwin", tool: "fabricate", path: "src/tools/rust-installer", mode: ToolBootstrap, is_optional_tool: false, source_type: Submodule, extra_features: [] }
+  85.380 (+ 29.15% /  292.947 =>  378.327) ToolBuild { compiler: Compiler { stage: 0, host: "x86_64-apple-darwin" }, target: "x86_64-apple-darwin", tool: "rustbook", path: "src/tools/rustbook", mode: ToolBootstrap, is_optional_tool: false, source_type: InTree, extra_features: [] }
+  88.275 (+104.80% /   84.233 =>  172.508) ToolBuild { compiler: Compiler { stage: 0, host: "x86_64-apple-darwin" }, target: "x86_64-apple-darwin", tool: "unstable-book-gen", path: "src/tools/unstable-book-gen", mode: ToolBootstrap, is_optional_tool: false, source_type: InTree, extra_features: [] }
+ 445.955 (+ 35.67% / 1250.219 => 1696.174) Rustc { target: "x86_64-apple-darwin", compiler: Compiler { stage: 0, host: "x86_64-apple-darwin" } }
+ 587.625 (+ 53.18% / 1104.933 => 1692.558) Rustc { target: "x86_64-apple-darwin", compiler: Compiler { stage: 1, host: "x86_64-apple-darwin" } }

Build time discrepancies of dist-x86_64-linux-alt

+  10.794 (+ 17.19% /   62.776 =>   73.570) CodegenBackend { compiler: Compiler { stage: 1, host: "x86_64-unknown-linux-gnu" }, target: "x86_64-unknown-linux-gnu", backend: "emscripten" }
+  11.570 (+ 15.77% /   73.344 =>   84.914) CodegenBackend { compiler: Compiler { stage: 1, host: "x86_64-unknown-linux-gnu" }, target: "x86_64-unknown-linux-gnu", backend: "llvm" }
+  15.973 (+ 23.07% /   69.243 =>   85.216) CodegenBackend { compiler: Compiler { stage: 0, host: "x86_64-unknown-linux-gnu" }, target: "x86_64-unknown-linux-gnu", backend: "emscripten" }
-  17.705 (- 16.81% /  105.308 =>   87.603) Std { compiler: Compiler { stage: 2, host: "x86_64-unknown-linux-gnu" }, target: "x86_64-unknown-linux-gnu" }
+  22.480 (+ 52.46% /   42.854 =>   65.334) ToolBuild { compiler: Compiler { stage: 0, host: "x86_64-unknown-linux-gnu" }, target: "x86_64-unknown-linux-gnu", tool: "fabricate", path: "src/tools/rust-installer", mode: ToolBootstrap, is_optional_tool: false, source_type: Submodule, extra_features: [] }
+  28.282 (+ 23.43% /  120.697 =>  148.979) Rustdoc { compiler: Compiler { stage: 2, host: "x86_64-unknown-linux-gnu" } }
+  29.662 (+ 24.48% /  121.168 =>  150.830) ToolBuild { compiler: Compiler { stage: 2, host: "x86_64-unknown-linux-gnu" }, target: "x86_64-unknown-linux-gnu", tool: "error_index_generator", path: "src/tools/error_index_generator", mode: ToolRustc, is_optional_tool: false, source_type: InTree, extra_features: [] }
+  29.855 (+ 45.25% /   65.980 =>   95.835) ToolBuild { compiler: Compiler { stage: 0, host: "x86_64-unknown-linux-gnu" }, target: "x86_64-unknown-linux-gnu", tool: "unstable-book-gen", path: "src/tools/unstable-book-gen", mode: ToolBootstrap, is_optional_tool: false, source_type: InTree, extra_features: [] }
+  32.225 (+ 33.03% /   97.576 =>  129.801) Std { target: "x86_64-unknown-linux-gnu", compiler: Compiler { stage: 1, host: "x86_64-unknown-linux-gnu" } }
-  37.837 (-  9.26% /  408.716 =>  370.879) Extended { stage: 2, host: "x86_64-unknown-linux-gnu", target: "x86_64-unknown-linux-gnu" }
+  38.345 (+ 23.16% /  165.570 =>  203.915) ToolBuild { compiler: Compiler { stage: 1, host: "x86_64-unknown-linux-gnu" }, target: "x86_64-unknown-linux-gnu", tool: "rustfmt", path: "src/tools/rustfmt", mode: ToolRustc, is_optional_tool: true, source_type: Submodule, extra_features: [] }
-  42.658 (- 20.62% /  206.879 =>  164.221) Rustc { compiler: Compiler { stage: 2, host: "x86_64-unknown-linux-gnu" } }
+  51.647 (+ 29.56% /  174.699 =>  226.346) Rustc { stage: 2, target: "x86_64-unknown-linux-gnu" }
+  52.415 (+ 53.75% /   97.522 =>  149.937) Rustdoc { compiler: Compiler { stage: 1, host: "x86_64-unknown-linux-gnu" } }
+  53.498 (+ 20.85% /  256.526 =>  310.024) ToolBuild { compiler: Compiler { stage: 1, host: "x86_64-unknown-linux-gnu" }, target: "x86_64-unknown-linux-gnu", tool: "clippy-driver", path: "src/tools/clippy", mode: ToolRustc, is_optional_tool: true, source_type: Submodule, extra_features: [] }
+  59.658 (+ 18.24% /  327.024 =>  386.682) ToolBuild { compiler: Compiler { stage: 1, host: "x86_64-unknown-linux-gnu" }, target: "x86_64-unknown-linux-gnu", tool: "rls", path: "src/tools/rls", mode: ToolRustc, is_optional_tool: true, source_type: Submodule, extra_features: ["clippy"] }
+  67.197 (+ 13.77% /  487.916 =>  555.113) PlainSourceTarball
+  77.209 (+ 57.64% /  133.956 =>  211.165) Llvm { target: "x86_64-unknown-linux-gnu", emscripten: true }
+ 124.271 (+ 31.22% /  398.026 =>  522.297) ToolBuild { compiler: Compiler { stage: 1, host: "x86_64-unknown-linux-gnu" }, target: "x86_64-unknown-linux-gnu", tool: "cargo", path: "src/tools/cargo", mode: ToolRustc, is_optional_tool: false, source_type: Submodule, extra_features: [] }
+ 138.442 (+ 54.00% /  256.360 =>  394.802) ToolBuild { compiler: Compiler { stage: 0, host: "x86_64-unknown-linux-gnu" }, target: "x86_64-unknown-linux-gnu", tool: "rustbook", path: "src/tools/rustbook", mode: ToolBootstrap, is_optional_tool: false, source_type: InTree, extra_features: [] }
+ 203.954 (+ 18.84% / 1082.769 => 1286.723) Llvm { target: "x86_64-unknown-linux-gnu", emscripten: false }
+ 260.261 (+ 23.46% / 1109.413 => 1369.674) Rustc { target: "x86_64-unknown-linux-gnu", compiler: Compiler { stage: 1, host: "x86_64-unknown-linux-gnu" } }
+ 457.296 (+ 47.71% /  958.455 => 1415.751) Rustc { target: "x86_64-unknown-linux-gnu", compiler: Compiler { stage: 0, host: "x86_64-unknown-linux-gnu" } }

From this all I can really see is that everything is just blanket slower on Azure, namely compiling rustc. That makes sense since compiling rustc is quite parallel already. It also doesn't look like azure is doing anything Travis/AppVeyor isn't by accident, and furthermore they're configured the same. All-in-all I don't really see a great win for us to cash in on here.

[lnicola]

Your "chart" has a Python backtrace at the end of the table.

Two stupid questions:

• are LLVM assertions enabled on Travis and AppVeyor?
• do the Travis and Azure VMs have the same number of cores?

[alexcrichton]

Digging further into the build logs of MSVC it looks like we're building really big crates like Cargo and rustc-ap-syntax twice. That'll save 10-ish minutes of build time so I'm going to investigate why that's coming up and how to prevent it.

@lnicola it depends on the builder for whether assertions are enabled or not. There is a different number of cores between the services, where we're using 4 on Travis/AppVeyor and only 2 on Azure.

[alexcrichton]

We've since switched to azure, so I don't think this issue is serving much any more, so I'm going to close.