Add typenum to benchmark list

Kobzol · Kobzol · commit e66a275dabb2 · 2024-03-23T14:29:12.000+01:00
diff --git a/collector/compile-benchmarks/README.md b/collector/compile-benchmarks/README.md
@@ -35,8 +35,8 @@ They mostly consist of real-world crates.
 - **hyper-0.14.18**: A fairly large crate. Utilizes async/await, and used by
   many Rust programs. The crate uses cargo features to enable large portions of its
   structure and is built with `--features=client,http1,http2,server,stream`.
-- **image-0.24.1**: Basic image processing functions and methods for 
-  converting to and from various image formats. Used often in graphics 
+- **image-0.24.1**: Basic image processing functions and methods for
+  converting to and from various image formats. Used often in graphics
   programming.
 - **libc-0.2.124**: An interface to `libc`. Contains many declarations of
   types, constants, and functions, but relatively little normal code. Stresses
@@ -53,6 +53,8 @@ They mostly consist of real-world crates.
   built with `--features=stm32f410` to have faster benchmarking times.
 - **syn-1.0.89**: A library for parsing Rust code. An important part of the Rust
   ecosystem.
+- **typenum-1.17.0**: A library that encodes integer computation within the trait system. Serves as
+  a stress test for the trait solver, but at the same time it is also a very popular crate.
 - **unicode-normalization-0.1.19**: Unicode character composition and decomposition
   utilities. Uses huge `match` statements that stress the compiler in unusual
   ways.
@@ -74,16 +76,18 @@ compiler in interesting ways.
   caused [poor performance](https://github.com/rust-lang/rust/issues/32278) in
   the past.
 - **ctfe-stress-5**: A stress test for compile-time function evaluation.
-- **deeply-nested-multi**: A small program containing multiple examples 
+- **deeply-nested-multi**: A small program containing multiple examples
   ([one](https://github.com/rust-lang/rust/issues/38528),
   [two](https://github.com/rust-lang/rust/issues/72408),
   [three](https://github.com/rust-lang/rust/issues/75992))
   of code that caused exponential behavior in the past.
 - **deep-vector**: A test containing a single large vector of zeroes, which
   caused [poor performance](https://github.com/rust-lang/rust/issues/20936) in
   the past. Stresses macro expansion and type inference.
-- **derive**: A large number of simple structs with a `#[derive]` attribute for common built-in traits such as Copy and Debug.
-- **externs**: A large number of extern functions has caused [slowdowns in the past](https://github.com/rust-lang/rust/pull/78448).
+- **derive**: A large number of simple structs with a `#[derive]` attribute for common built-in
+  traits such as Copy and Debug.
+- **externs**: A large number of extern functions has
+  caused [slowdowns in the past](https://github.com/rust-lang/rust/pull/78448).
 - **helloworld-tiny**: A trivial program optimized with flags that should reduce binary size.
   Gives a lower bound on compiled binary size.
 - **issue-46449**: A small program that caused [poor
@@ -96,7 +100,7 @@ compiler in interesting ways.
 - **many-assoc-items**: Contains a struct with many associated items, which
   caused [quadratic behavior](https://github.com/rust-lang/rust/issues/68957)
   in the past.
-- **match-stress**: Contains examples 
+- **match-stress**: Contains examples
   (one involving [a huge enum](https://github.com/rust-lang/rust/issues/7462),
   one involving
   [`exhaustive_patterns`](https://github.com/rust-lang/rust/pull/79394)) of
@@ -169,90 +173,92 @@ Rust code being written today.
   crate, and secondary if it's a stress test or intended to catch specific
   regressions.
 - If it's a third-party crate:
-  - If you are keen: talk with a maintainer of the crate to see if there is
-    anything we should be aware of when using this crate as a compile-time
-    benchmark.
-  - Look at [crates.io](https://crates.io) to find the latest (non-prerelease) version.
-  - Download it with `collector download -c $CATEGORY -a $ARTIFACT crate $NAME $VERSION`.
-    The `$CATEGORY` is probably `primary`. `$ARTIFACT` is either `library` or `binary`, depending
-    on what kind of artifact does the benchmark build.
+    - If you are keen: talk with a maintainer of the crate to see if there is
+      anything we should be aware of when using this crate as a compile-time
+      benchmark.
+    - Look at [crates.io](https://crates.io) to find the latest (non-prerelease) version.
+    - Download it with `collector download -c $CATEGORY -a $ARTIFACT crate $NAME $VERSION`.
+      The `$CATEGORY` is probably `primary`. `$ARTIFACT` is either `library` or `binary`, depending
+      on what kind of artifact does the benchmark build.
 - It makes it easier for reviewers if you split things into two commits.
 - In the first commit, just add the code for the entire benchmark.
-  - Do this by doing `git add` on the new directory.
-  - There is no need to remove seemingly unnecessary files such as
-    documentation or CI configuration.
+    - Do this by doing `git add` on the new directory.
+    - There is no need to remove seemingly unnecessary files such as
+      documentation or CI configuration.
 - In the second commit, do everything else.
-  - Add `[workspace]` to the very bottom of the benchmark's `Cargo.toml`, if
-    doesn't already have a `[workspace]` section. This means commands like
-    `cargo build` will work within the benchmark directory.
-  - Add any necessary stuff to the `perf-config.json` file.
-    - If the benchmark is a sub-crate within a top-level crate, you'll need a
-      `"cargo_toml"` entry.
-    - If you get a "non-wrapped rustc" error when running it, you'll need a
-      `"touch_file"` entry.
-    - See [`collector/src/benchmark/mod.rs`](https://github.com/rust-lang/rustc-perf/blob/12cb796f8a932a891b385ba23a36d78a2867ace1/collector/src/benchmark/mod.rs#L24-L27) for a complete reference.
-  - Consider adding one or more `N-*.patch` files for the `IncrPatched`
-    scenario.
-    - If it's a primary benchmark, you should definitely do this.
-    - These usually consist of a patch that adds a single
-      `println!("testing");` statement somewhere.
-    - Creating the patch against what you've committed so far might be useful.
-      Use `git diff` from the repository root, or `git diff --relative` within
-      the benchmark directory. Copy the output into the `N-*.patch` file.
-    - Do a test run with an `IncrPatched` scenario to make sure the patch
-      applies correctly, e.g. `target/release/collector bench_local +nightly
-      --id Test --profiles=Check --scenarios=IncrPatched
-      --include=$NEW_BENCHMARK`
-  - Add the new entry to `collector/compile-benchmarks/README.md`.
-  - `git add` the `Cargo.lock` file, if it's not already part of the
-    benchmark's committed code.
-    - If the benchmark has a `.gitignore` file that contains `Cargo.lock`,
-      you'll need to comment out that line so that `Cargo.lock` gets uploaded
-      in the PR.
+    - Add `[workspace]` to the very bottom of the benchmark's `Cargo.toml`, if
+      doesn't already have a `[workspace]` section. This means commands like
+      `cargo build` will work within the benchmark directory.
+    - Add any necessary stuff to the `perf-config.json` file.
+        - If the benchmark is a sub-crate within a top-level crate, you'll need a
+          `"cargo_toml"` entry.
+        - If you get a "non-wrapped rustc" error when running it, you'll need a
+          `"touch_file"` entry.
+        -
+        See [`collector/src/benchmark/mod.rs`](https://github.com/rust-lang/rustc-perf/blob/12cb796f8a932a891b385ba23a36d78a2867ace1/collector/src/benchmark/mod.rs#L24-L27)
+        for a complete reference.
+    - Consider adding one or more `N-*.patch` files for the `IncrPatched`
+      scenario.
+        - If it's a primary benchmark, you should definitely do this.
+        - These usually consist of a patch that adds a single
+          `println!("testing");` statement somewhere.
+        - Creating the patch against what you've committed so far might be useful.
+          Use `git diff` from the repository root, or `git diff --relative` within
+          the benchmark directory. Copy the output into the `N-*.patch` file.
+        - Do a test run with an `IncrPatched` scenario to make sure the patch
+          applies correctly, e.g. `target/release/collector bench_local +nightly
+          --id Test --profiles=Check --scenarios=IncrPatched
+          --include=$NEW_BENCHMARK`
+    - Add the new entry to `collector/compile-benchmarks/README.md`.
+    - `git add` the `Cargo.lock` file, if it's not already part of the
+      benchmark's committed code.
+        - If the benchmark has a `.gitignore` file that contains `Cargo.lock`,
+          you'll need to comment out that line so that `Cargo.lock` gets uploaded
+          in the PR.
 - Consider the benchmarking time for the benchmark.
-  - First, measure the entire compilation time with something like this, by
-    doing this within the benchmark directory is good:
-    ```
-    CARGO_INCREMENTAL=0 cargo check ; cargo clean
-    CARGO_INCREMENTAL=0 cargo build ; cargo clean
-    CARGO_INCREMENTAL=0 cargo build --release ; cargo clean
-    ```
-  - Second, compare the final crate time with these commands:
-    ```
-    target/release/collector bench_local +nightly --id Test \
-      --profiles=Check,Debug,Opt --scenarios=Full --include=$NEW,helloworld
-    target/release/site results.db
-    ```
-    (See [here](../../site/README.md) for instructions on how to build the website).
-    Then switch to wall-times, compare `Test` against itself, and toggle the
-    "Show non-relevant results"/"Display raw data" check boxes to make sure it
-    hasn't changed drastically.
-    - E.g. `futures` was changed so it's just a facade for a bunch of
-      sub-crates, and the final crate time became very similar to `helloworld`,
-      which wasn't interesting.
+    - First, measure the entire compilation time with something like this, by
+      doing this within the benchmark directory is good:
+      ```
+      CARGO_INCREMENTAL=0 cargo check ; cargo clean
+      CARGO_INCREMENTAL=0 cargo build ; cargo clean
+      CARGO_INCREMENTAL=0 cargo build --release ; cargo clean
+      ```
+    - Second, compare the final crate time with these commands:
+      ```
+      target/release/collector bench_local +nightly --id Test \
+        --profiles=Check,Debug,Opt --scenarios=Full --include=$NEW,helloworld
+      target/release/site results.db
+      ```
+      (See [here](../../site/README.md) for instructions on how to build the website).
+      Then switch to wall-times, compare `Test` against itself, and toggle the
+      "Show non-relevant results"/"Display raw data" check boxes to make sure it
+      hasn't changed drastically.
+        - E.g. `futures` was changed so it's just a facade for a bunch of
+          sub-crates, and the final crate time became very similar to `helloworld`,
+          which wasn't interesting.
 - File a PR, including the two sets of timing measurements in the description.
 
 ## Remove a benchmark
 
 - It makes it easier for reviewers if you split things into two commits.
 - In the first commit just remove the old code.
-  - Do this with `git rm -r` on the directory.
+    - Do this with `git rm -r` on the directory.
 - In the second commit do everything else.
-  - Remove the entry from `collector/compile-benchmarks/README.md`.
-  - `git grep` for occurrences of the old benchmark name (e.g. in
-    `.github/workflows/ci.yml` or `ci/check-*.sh`) and see if anything needs
-    changing... usually not.
+    - Remove the entry from `collector/compile-benchmarks/README.md`.
+    - `git grep` for occurrences of the old benchmark name (e.g. in
+      `.github/workflows/ci.yml` or `ci/check-*.sh`) and see if anything needs
+      changing... usually not.
 - File a PR.
 
 ## Update a benchmark
 
 - Do this in two steps.
-  - First add the new version of the benchmark. Compare the benchmarking time
-    of the two versions to make sure nothing outrageous has happened. Once the
-    PR is merged, make sure it's running correctly.
-  - Second, remove the old version of the benchmark.
-  Doing it in two steps ensures we have continuity of profiling coverage in the
-  case where something goes wrong.
+    - First add the new version of the benchmark. Compare the benchmarking time
+      of the two versions to make sure nothing outrageous has happened. Once the
+      PR is merged, make sure it's running correctly.
+    - Second, remove the old version of the benchmark.
+      Doing it in two steps ensures we have continuity of profiling coverage in the
+      case where something goes wrong.
 - Compare the benchmarking time of the two versions.
 - When adding the new version, for `perf-config.json` and the `N-*.patch`
   files, use the corresponding files for the old version as a starting point.
@@ -266,15 +272,17 @@ benchmarks in rustc-perf are verbatim copies of third-party crates. We
 periodically do a mass update of these benchmarks.
 
 Benefits of this approach:
+
 - We ensure we are measuring compilation of the crates most people are using.
   This is most relevant for popular crates.
 - We get coverage of newer language features.
 
 Costs of this approach:
+
 - It takes time and effort.
 - We lose some data continuity.
-  - But the stable set of benchmarks used for the dashboard are not affected,
-    and they provide the greatest continuity.
+    - But the stable set of benchmarks used for the dashboard are not affected,
+      and they provide the greatest continuity.
 - If the code hasn't changed much, it won't have much effect.
 
 ## Update policy
@@ -297,5 +305,6 @@ Costs of this approach:
   maintainers.
 
 History:
+
 - The first mass update of third-party crates occurred in [March/April
   2022](https://hackmd.io/d9uE7qgtTWKDLivy0uoVQw).
