Enable benchmarks to compare HEAD with a previous release #6818
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Align the junit/test output path with SBT's. This allows you to
compile in SBT and trigger a test execution (including debugging)
in IntelliJ with a run configuration that has "build" removed as
a pre-run ste.
- Add the benchmarks subproject. This gives autocomplete and build
within IntelliJ. Benchmarks still need to be run under SBT, however,
to let sbt-jmh do the code generation.
- Add some more auto-imports to our hand-rolled definitions of the
scala-build project, to eliminate some highlighting errors in
build.sbt
For instance: ``` $ for V in 2.12.6 ""; do label=$V; [[ "$V" == "" ]] && label=2.13.x; (set -x; sbt -Dbenchmark.scala.version="$V" 'bench/jmh:run scala.collection.mutable.HashMapBenchmark.* -psize=1000 -f1 -rf json -rff /tmp/'$label'-result.json'); done + sbt -Dbenchmark.scala.version=2.12.6 'bench/jmh:run scala.collection.mutable.HashMapBenchmark.* -psize=1000 -f1 -rf json -rff /tmp/2.12.6-result.json' [info] Loading global plugins from /Users/jz/.sbt/0.13/plugins [info] Loading project definition from /Users/jz/code/scala/project/project [info] Loading project definition from /Users/jz/code/scala/project [info] Resolving key references (11225 settings) ... [info] *** Welcome to the sbt build definition for Scala! *** [info] Check README.md for more information. [info] Running org.openjdk.jmh.Main scala.collection.mutable.HashMapBenchmark.* -psize=1000 -f1 -rf json -rff /tmp/2.12.6-result.json [info] # JMH version: 1.20 [info] # VM version: JDK 1.8.0_172, VM 25.172-b11 [info] # VM invoker: /Users/jz/.jabba/jdk/1.8.172/Contents/Home/jre/bin/java [info] # VM options: <none> [info] # Warmup: 10 iterations, 1 s each [info] # Measurement: 10 iterations, 1 s each [info] # Timeout: 10 min per iteration [info] # Threads: 1 thread, will synchronize iterations [info] # Benchmark mode: Average time, time/op [info] # Benchmark: scala.collection.mutable.HashMapBenchmark.get [info] # Parameters: (size = 1000) [info] [info] # Run progress: 0.00% complete, ETA 00:01:00 [info] # Fork: 1 of 1 [info] # Warmup Iteration 1: 99768.671 ns/op [info] # Warmup Iteration 2: 95825.105 ns/op [info] # Warmup Iteration 3: 75081.823 ns/op [info] # Warmup Iteration 4: 72828.836 ns/op [info] # Warmup Iteration 5: 78498.108 ns/op [info] # Warmup Iteration 6: 77529.326 ns/op [info] # Warmup Iteration 7: 73642.270 ns/op [info] # Warmup Iteration 8: 74381.108 ns/op [info] # Warmup Iteration 9: 85251.408 ns/op [info] # Warmup Iteration 10: 80493.922 ns/op [info] Iteration 1: 81787.081 ns/op [info] Iteration 2: 78877.519 ns/op [info] Iteration 3: 80313.631 ns/op [info] Iteration 4: 87345.949 ns/op [info] Iteration 5: 82615.938 ns/op [info] Iteration 6: 78870.927 ns/op [info] Iteration 7: 77012.624 ns/op [info] Iteration 8: 76206.510 ns/op [info] Iteration 9: 77893.311 ns/op [info] Iteration 10: 74737.901 ns/op [info] [info] [info] Result "scala.collection.mutable.HashMapBenchmark.get": [info] 79566.139 ±(99.9%) 5524.968 ns/op [Average] [info] (min, avg, max) = (74737.901, 79566.139, 87345.949), stdev = 3654.424 [info] CI (99.9%): [74041.171, 85091.107] (assumes normal distribution) [info] [info] [info] # JMH version: 1.20 [info] # VM version: JDK 1.8.0_172, VM 25.172-b11 [info] # VM invoker: /Users/jz/.jabba/jdk/1.8.172/Contents/Home/jre/bin/java [info] # VM options: <none> [info] # Warmup: 10 iterations, 1 s each [info] # Measurement: 10 iterations, 1 s each [info] # Timeout: 10 min per iteration [info] # Threads: 1 thread, will synchronize iterations [info] # Benchmark mode: Average time, time/op [info] # Benchmark: scala.collection.mutable.HashMapBenchmark.getOrElseUpdate [info] # Parameters: (size = 1000) [info] [info] # Run progress: 33.33% complete, ETA 00:00:41 [info] # Fork: 1 of 1 [info] # Warmup Iteration 1: 58258.214 ns/op [info] # Warmup Iteration 2: 51326.789 ns/op [info] # Warmup Iteration 3: 52674.458 ns/op [info] # Warmup Iteration 4: 54007.246 ns/op [info] # Warmup Iteration 5: 56024.431 ns/op [info] # Warmup Iteration 6: 55215.232 ns/op [info] # Warmup Iteration 7: 58845.959 ns/op [info] # Warmup Iteration 8: 55399.081 ns/op [info] # Warmup Iteration 9: 53096.872 ns/op [info] # Warmup Iteration 10: 51450.088 ns/op [info] Iteration 1: 50830.953 ns/op [info] Iteration 2: 51113.487 ns/op [info] Iteration 3: 53496.845 ns/op [info] Iteration 4: 52221.525 ns/op [info] Iteration 5: 50491.810 ns/op [info] Iteration 6: 50331.400 ns/op [info] Iteration 7: 51159.254 ns/op [info] Iteration 8: 50845.059 ns/op [info] Iteration 9: 50981.678 ns/op [info] Iteration 10: 51362.981 ns/op [info] [info] [info] Result "scala.collection.mutable.HashMapBenchmark.getOrElseUpdate": [info] 51283.499 ±(99.9%) 1412.042 ns/op [Average] [info] (min, avg, max) = (50331.400, 51283.499, 53496.845), stdev = 933.978 [info] CI (99.9%): [49871.457, 52695.541] (assumes normal distribution) [info] [info] [info] # JMH version: 1.20 [info] # VM version: JDK 1.8.0_172, VM 25.172-b11 [info] # VM invoker: /Users/jz/.jabba/jdk/1.8.172/Contents/Home/jre/bin/java [info] # VM options: <none> [info] # Warmup: 10 iterations, 1 s each [info] # Measurement: 10 iterations, 1 s each [info] # Timeout: 10 min per iteration [info] # Threads: 1 thread, will synchronize iterations [info] # Benchmark mode: Average time, time/op [info] # Benchmark: scala.collection.mutable.HashMapBenchmark.put [info] # Parameters: (size = 1000) [info] [info] # Run progress: 66.67% complete, ETA 00:00:20 [info] # Fork: 1 of 1 [info] # Warmup Iteration 1: 78554.352 ns/op [info] # Warmup Iteration 2: 70268.328 ns/op [info] # Warmup Iteration 3: 65631.044 ns/op [info] # Warmup Iteration 4: 65859.027 ns/op [info] # Warmup Iteration 5: 65608.479 ns/op [info] # Warmup Iteration 6: 65401.858 ns/op [info] # Warmup Iteration 7: 65539.139 ns/op [info] # Warmup Iteration 8: 65762.776 ns/op [info] # Warmup Iteration 9: 66208.764 ns/op [info] # Warmup Iteration 10: 66564.731 ns/op [info] Iteration 1: 66780.604 ns/op [info] Iteration 2: 67820.270 ns/op [info] Iteration 3: 66814.685 ns/op [info] Iteration 4: 67286.592 ns/op [info] Iteration 5: 66502.141 ns/op [info] Iteration 6: 67057.898 ns/op [info] Iteration 7: 66419.030 ns/op [info] Iteration 8: 66772.388 ns/op [info] Iteration 9: 66567.026 ns/op [info] Iteration 10: 66488.257 ns/op [info] [info] [info] Result "scala.collection.mutable.HashMapBenchmark.put": [info] 66850.889 ±(99.9%) 658.041 ns/op [Average] [info] (min, avg, max) = (66419.030, 66850.889, 67820.270), stdev = 435.253 [info] CI (99.9%): [66192.848, 67508.930] (assumes normal distribution) [info] [info] [info] # Run complete. Total time: 00:01:02 [info] [info] Benchmark (size) Mode Cnt Score Error Units [info] HashMapBenchmark.get 1000 avgt 10 79566.139 ± 5524.968 ns/op [info] HashMapBenchmark.getOrElseUpdate 1000 avgt 10 51283.499 ± 1412.042 ns/op [info] HashMapBenchmark.put 1000 avgt 10 66850.889 ± 658.041 ns/op [info] [info] Benchmark result is saved to /tmp/2.12.6-result.json [success] Total time: 63 s, completed 19/06/2018 6:56:41 PM + sbt -Dbenchmark.scala.version= 'bench/jmh:run scala.collection.mutable.HashMapBenchmark.* -psize=1000 -f1 -rf json -rff /tmp/2.13.x-result.json' [info] Loading global plugins from /Users/jz/.sbt/0.13/plugins [info] Loading project definition from /Users/jz/code/scala/project/project [info] Loading project definition from /Users/jz/code/scala/project [info] Resolving key references (11230 settings) ... [info] *** Welcome to the sbt build definition for Scala! *** [info] Check README.md for more information. Processing 176 classes from /Users/jz/code/scala/test/benchmarks/target/scala-2.13.0-M4/classes with "reflection" generator Writing out Java source to /Users/jz/code/scala/test/benchmarks/target/scala-2.13.0-M4/src_managed/jmh and resources to /Users/jz/code/scala/test/benchmarks/target/scala-2.13.0-M4/resource_managed/jmh [info] Compiling 4 Java sources to /Users/jz/code/scala/test/benchmarks/target/scala-2.13.0-M4/classes... [info] Running org.openjdk.jmh.Main scala.collection.mutable.HashMapBenchmark.* -psize=1000 -f1 -rf json -rff /tmp/2.13.x-result.json [info] # JMH version: 1.20 [info] # VM version: JDK 1.8.0_172, VM 25.172-b11 [info] # VM invoker: /Users/jz/.jabba/jdk/1.8.172/Contents/Home/jre/bin/java [info] # VM options: <none> [info] # Warmup: 10 iterations, 1 s each [info] # Measurement: 10 iterations, 1 s each [info] # Timeout: 10 min per iteration [info] # Threads: 1 thread, will synchronize iterations [info] # Benchmark mode: Average time, time/op [info] # Benchmark: scala.collection.mutable.HashMapBenchmark.get [info] # Parameters: (size = 1000) [info] [info] # Run progress: 0.00% complete, ETA 00:01:00 [info] # Fork: 1 of 1 [info] # Warmup Iteration 1: 95093.592 ns/op [info] # Warmup Iteration 2: 91922.469 ns/op [info] # Warmup Iteration 3: 83172.419 ns/op [info] # Warmup Iteration 4: 81480.126 ns/op [info] # Warmup Iteration 5: 81617.458 ns/op [info] # Warmup Iteration 6: 81772.240 ns/op [info] # Warmup Iteration 7: 80737.285 ns/op [info] # Warmup Iteration 8: 82445.206 ns/op [info] # Warmup Iteration 9: 83241.661 ns/op [info] # Warmup Iteration 10: 81271.995 ns/op [info] Iteration 1: 82186.458 ns/op [info] Iteration 2: 82074.212 ns/op [info] Iteration 3: 82544.867 ns/op [info] Iteration 4: 82158.709 ns/op [info] Iteration 5: 82468.038 ns/op [info] Iteration 6: 81986.508 ns/op [info] Iteration 7: 81196.385 ns/op [info] Iteration 8: 81456.118 ns/op [info] Iteration 9: 81993.954 ns/op [info] Iteration 10: 82489.535 ns/op [info] [info] [info] Result "scala.collection.mutable.HashMapBenchmark.get": [info] 82055.478 ±(99.9%) 663.246 ns/op [Average] [info] (min, avg, max) = (81196.385, 82055.478, 82544.867), stdev = 438.696 [info] CI (99.9%): [81392.232, 82718.724] (assumes normal distribution) [info] [info] [info] # JMH version: 1.20 [info] # VM version: JDK 1.8.0_172, VM 25.172-b11 [info] # VM invoker: /Users/jz/.jabba/jdk/1.8.172/Contents/Home/jre/bin/java [info] # VM options: <none> [info] # Warmup: 10 iterations, 1 s each [info] # Measurement: 10 iterations, 1 s each [info] # Timeout: 10 min per iteration [info] # Threads: 1 thread, will synchronize iterations [info] # Benchmark mode: Average time, time/op [info] # Benchmark: scala.collection.mutable.HashMapBenchmark.getOrElseUpdate [info] # Parameters: (size = 1000) [info] [info] # Run progress: 33.33% complete, ETA 00:00:41 [info] # Fork: 1 of 1 [info] # Warmup Iteration 1: 65972.824 ns/op [info] # Warmup Iteration 2: 63418.276 ns/op [info] # Warmup Iteration 3: 63064.944 ns/op [info] # Warmup Iteration 4: 63109.554 ns/op [info] # Warmup Iteration 5: 63893.090 ns/op [info] # Warmup Iteration 6: 63305.788 ns/op [info] # Warmup Iteration 7: 63565.730 ns/op [info] # Warmup Iteration 8: 63522.491 ns/op [info] # Warmup Iteration 9: 64163.203 ns/op [info] # Warmup Iteration 10: 63304.608 ns/op [info] Iteration 1: 63700.486 ns/op [info] Iteration 2: 63476.873 ns/op [info] Iteration 3: 64419.062 ns/op [info] Iteration 4: 63256.231 ns/op [info] Iteration 5: 63742.293 ns/op [info] Iteration 6: 63802.757 ns/op [info] Iteration 7: 63869.556 ns/op [info] Iteration 8: 63381.295 ns/op [info] Iteration 9: 63851.647 ns/op [info] Iteration 10: 63431.670 ns/op [info] [info] [info] Result "scala.collection.mutable.HashMapBenchmark.getOrElseUpdate": [info] 63693.187 ±(99.9%) 504.197 ns/op [Average] [info] (min, avg, max) = (63256.231, 63693.187, 64419.062), stdev = 333.495 [info] CI (99.9%): [63188.990, 64197.384] (assumes normal distribution) [info] [info] [info] # JMH version: 1.20 [info] # VM version: JDK 1.8.0_172, VM 25.172-b11 [info] # VM invoker: /Users/jz/.jabba/jdk/1.8.172/Contents/Home/jre/bin/java [info] # VM options: <none> [info] # Warmup: 10 iterations, 1 s each [info] # Measurement: 10 iterations, 1 s each [info] # Timeout: 10 min per iteration [info] # Threads: 1 thread, will synchronize iterations [info] # Benchmark mode: Average time, time/op [info] # Benchmark: scala.collection.mutable.HashMapBenchmark.put [info] # Parameters: (size = 1000) [info] [info] # Run progress: 66.67% complete, ETA 00:00:20 [info] # Fork: 1 of 1 [info] # Warmup Iteration 1: 78231.854 ns/op [info] # Warmup Iteration 2: 69636.700 ns/op [info] # Warmup Iteration 3: 65993.942 ns/op [info] # Warmup Iteration 4: 65843.085 ns/op [info] # Warmup Iteration 5: 66153.593 ns/op [info] # Warmup Iteration 6: 65965.618 ns/op [info] # Warmup Iteration 7: 65835.985 ns/op [info] # Warmup Iteration 8: 66168.597 ns/op [info] # Warmup Iteration 9: 66138.030 ns/op [info] # Warmup Iteration 10: 66515.731 ns/op [info] Iteration 1: 66544.956 ns/op [info] Iteration 2: 66006.530 ns/op [info] Iteration 3: 66168.568 ns/op [info] Iteration 4: 66406.033 ns/op [info] Iteration 5: 66274.727 ns/op [info] Iteration 6: 66209.624 ns/op [info] Iteration 7: 66271.907 ns/op [info] Iteration 8: 66535.960 ns/op [info] Iteration 9: 66748.752 ns/op [info] Iteration 10: 67022.122 ns/op [info] [info] [info] Result "scala.collection.mutable.HashMapBenchmark.put": [info] 66418.918 ±(99.9%) 456.722 ns/op [Average] [info] (min, avg, max) = (66006.530, 66418.918, 67022.122), stdev = 302.093 [info] CI (99.9%): [65962.196, 66875.639] (assumes normal distribution) [info] [info] [info] # Run complete. Total time: 00:01:01 [info] [info] Benchmark (size) Mode Cnt Score Error Units [info] HashMapBenchmark.get 1000 avgt 10 82055.478 ± 663.246 ns/op [info] HashMapBenchmark.getOrElseUpdate 1000 avgt 10 63693.187 ± 504.197 ns/op [info] HashMapBenchmark.put 1000 avgt 10 66418.918 ± 456.722 ns/op [info] [info] Benchmark result is saved to /tmp/2.13.x-result.json [success] Total time: 68 s, completed 19/06/2018 6:57:56 PM /code/scala on topic/benchpress* $ open http://jmh.morethan.io/ # drag /tmp/*.result.json into the drop-zone in the UI ``` Showing a regression in `getOrElseUpdate`: http://jmh.morethan.io/?sources=https://gist.githubusercontent.com/retronym/a8d08ba76016f338f625e3bb9b67ebfa/raw/93ec9d3e109be4c695c958f95385bd18a8241268/2.13.x-result.json,https://gist.githubusercontent.com/retronym/a8d08ba76016f338f625e3bb9b67ebfa/raw/93ec9d3e109be4c695c958f95385bd18a8241268/2.12.6-result.json
retronym
changed the title
|
I think the performance change in |
retronym
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For instance:
The results can be visualized with jmh-visualizer
Locally:
Or by uploading the results (e.g. as gists) and pointing the visualizer at the URLs
This shows a regression in
mutable.HashMap.getOrElseUpdate.