Profile + Tune compiler

[retronym]

After bootstrapping with the all the latest and greatest in 2.12 (new optimizer, indy functions, no impl classes), a quick test shows performance is about on par with 2.11.8. I tested this by compiling src/library with both compilers.

Analysing the YourKit profiles of the new compiler does show some small opportunities for improvements.

• Name related operations, addressed by Micro optimise Symbol#fullName scala#4954 (not yet merged to 2.12.x). This needs to be merged forwards to 2.12.x
• Initialization of TyperDiagnostics#deadCode (which is itself dead code outside of -Ydead-code !)
• Initialiation of object ArrayInstantiation on every typedSelect (no need for that to be nested where it is, move to TreeInfo.
• Initalization of Reporter#ERROR. We create a lot more Reporter instances these times, we should try to be less allocation hungry in the common case when hasErrors is false.
• Consider changing reflect.internal.util.WeakHashSet to use .equals rather than == (to avoid type tests in BoxesRuntimes.equals.

I've got additional ideas and tools that I collected previously here: https://gist.github.com/retronym/86ec6ad9ccd2c22f6148

[retronym]

/cc @gkossakowski @lrytz

[lrytz]

in my testing it looks like 2.12.0-M2 (maybe already M1?) is slower than 2.11.8, see https://docs.google.com/spreadsheets/d/12-GILZectCueVLeyC0HjOlnlQRZPBcTADGf9sVmGAW0/edit#gid=37831420

the slowdown in "2.12.x" (in the sheet, 2.12.0-M5) is identified, blog post with analysis coming up.

[retronym]

@lrytz Are you testing cold or warm performance?

[DarkDimius]

see also scala/scala3#1320 where I compile a silly file with cold performance.

[lrytz]

good question, i was looking at cold performance

for i in {1..10}; do time /Users/luc/scala/scala-2.12.0-M5/bin/scalac @files; done

[lrytz]

we should test both

[retronym]

There is a simple harness for repeated compilation in:

% scala 'scala.tools.nsc.MainBench' sandbox/test.scala

But we probably should build this into our new JMH suite to have a standard way to control iterations, number of forks, warmups, convergences etc/

[retronym]

package scala.tools.nsc

import java.nio.charset.Charset
import java.nio.file.{Files, Path}

import org.openjdk.jmh.annotations.{Benchmark, Scope, Setup, State, _}

import scala.reflect.io.AbstractFile

@State(Scope.
  Thread)
class CompilerBenchmark {

  @Param(Array("-version"))
  var compilerArgs: String = _

  @Setup
  def setup {
  }

  @Benchmark
  def compile: Boolean = {
    val driver = new scala.tools.nsc.MainClass {
      override def processSettingsHook(): Boolean = {
        settings.usejavacp.value = true
        true
      }
    }
    driver.process(compilerArgs.split(" +"))
    true
  }
}

args.txt:

-nowarn
-language:_
/code/better-files/core/src/main//scala/better/files/Cmds.scala
/code/better-files/core/src/main//scala/better/files/File.scala
/code/better-files/core/src/main//scala/better/files/Implicits.scala
/code/better-files/core/src/main//scala/better/files/package.scala
/code/better-files/core/src/main//scala/better/files/Scanner.scala
/code/better-files/core/src/main//scala/better/files/ThreadBackedFileMonitor.scala

2.11.8

% sbt 'set List(scalacOptions := Nil, scalaHome := None, scalaVersion := "2.11.8")' "jmh:run -p compilerArgs=@/code/better-files/args.txt -f 1 -wi 10 -i 10 -bm avgt scala.tools.nsc.CompilerBenchmark"


Benchmark                                (compilerArgs)  Mode  Cnt  Score   Error  Units
CompilerBenchmark.compile  @/code/better-files/args.txt  avgt   10  0.621 ± 0.067   s/op

2.12.0-M4

% sbt 'set List(scalacOptions := Nil, scalaHome := None, scalaVersion := "2.12.0-M4")' "jmh:run -p compilerArgs=@/code/better-files/args.txt -f 1 -wi 10 -i 10 -bm avgt scala.tools.nsc.CompilerBenchmark"

Benchmark                                (compilerArgs)  Mode  Cnt  Score   Error  Units
CompilerBenchmark.compile  @/code/better-files/args.txt  avgt   10  0.610 ± 0.042   s/op
[success] Total time: 38 s, completed 05/07/2016 5:57:58 PM

2.12.0-M5:

% sbt set 'set List(scalacOptions := Nil, scalaHome := None, scalaVersion := "2.12.0-M5")' "jmh:run -p compilerArgs=@/code/better-files/args.txt -f 1 -wi 10 -i 10 -bm avgt scala.tools.nsc.CompilerBenchmark"
Benchmark                                (compilerArgs)  Mode  Cnt  Score   Error  Units
CompilerBenchmark.compile  @/code/better-files/args.txt  avgt   10  0.611 ± 0.055   s/op

I ran this by piggybacking on the library benchmark suite in test/benchmarks, but it occurs to me that we should create a separate SBT project for compiler benchmarks so that the benchemark code can easily cross compile against 2.11 and 2.12.

[lrytz]

maybe we can build on top of my benchmarks repo? https://github.com/lrytz/benchmarks

[lrytz]

by the way, what is 2.12.x? the current branch, i.e., M5? given the known regression, it would be interesting to compare against M4.