ConcurrentModificationException in the compiler

[noresttherein]

Scala 2.13.10

Now this is something I haven't seen. StackOverflowError sure, half a dozen a day, I can understand them. But this?

java.util.ConcurrentModificationException: mutation occurred during iteration
	at scala.collection.mutable.MutationTracker$.checkMutations(MutationTracker.scala:43)
	at scala.collection.mutable.MutationTracker$CheckedIterator.hasNext(MutationTracker.scala:59)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:575)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:573)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:933)
	at scala.tools.nsc.typechecker.Analyzer$typerFactory$TyperPhase.apply(Analyzer.scala:120)
	at scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:467)
	at scala.tools.nsc.typechecker.Analyzer$typerFactory$TyperPhase.run(Analyzer.scala:106)
	at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1530)
	at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1514)
	at scala.tools.nsc.Global$Run.compileSources(Global.scala:1506)
	at scala.tools.nsc.Global$Run.compileFiles(Global.scala:1619)
	at xsbt.CachedCompiler0.run(CompilerBridge.scala:163)
	at xsbt.CachedCompiler0.run(CompilerBridge.scala:134)
	at xsbt.CompilerBridge.run(CompilerBridge.scala:39)
	at sbt.internal.inc.AnalyzingCompiler.compile(AnalyzingCompiler.scala:91)

I haven't a faintest idea in what class, file or even package this happens, so I have no way of extracting it. Maybe it will tell something to someone. Any ideas of how to look for clues? I have about every logging option enabled which doesn't throw an AssertionError, or NullPointerException itself. Unfortunately, the last line from -Ylog:0-100 seems to be from a successfully processed file (I got a summary of warnings for the file), so I am at a loss.

[som-snytt]

Probably it's the post-typer checks, unit.toCheck, that is getting updated during iteration.

There was just an unrelated improvement in that mechanism.

[SethTisue]

ha, I haven't seen this one before either.

the code in question (in Analyzer.scala), for (workItem <- unit.toCheck) workItem(), where toCheck is a ListBuffer of thunks and there are various sites elsewhere where we do unit.toCheck += ..., fails thanks to scala/scala#9174 (merged in Scala 2.13.4)

[som-snytt]

I get to say jinx before Seth does?

[SethTisue]

I owe @som-snytt a Coke

it looks like we could make the code in question safe pretty easily, and without noticeably impacting performance (it's very much an outer loop), by replacing

for (workItem <- unit.toCheck) workItem()

with something like

while (unit.toCheck.nonEmpty) {
  val items = unit.toCheck.toList
  unit.toCheck.clear()
  for (workItem <- items) workItem()
}

that kind of thing

I am swamped this week, perhaps someone else would like to jump on it first?

[som-snytt]

These are checks run (currently) after a file is typechecked (and just before unused warnings), so it should (perhaps) be obvious from debug output which file was just typechecked. A reproduction would be nice. I'll volunteer the easy fix as proposed.

[noresttherein]

Yes, I do have output giving a strong indication of what has been just typechecked. But I have no idea what can I do to improve this issue on my side? Of course, as far as reproduction goes, I can link a commit in my project which fails, but

1. It's a fairly large project
2. Very complex for the type checker in general
3. And there are currently 100+ legitimate compiler issues on top of that one
   So I did not think anyone would be interested, especially that you seem to zoned in on the issue pretty well without it.

[noresttherein]

Can I somehow compile just those two files against current 2.13.10 and put them in bootclasspath?

[som-snytt]

I don't know the simplest way to put a couple of class files on sbt's class path, if that is the question.

Usually I wind up publishing a local jar to include in the build.

I will try to enumerate when it adds "post-typer" work items and which might incur such tasks recursively. I think if it tries to add a work item after typer has run, it should just execute it immediately. (I tried the enumeration yesterday but fell asleep.)

[noresttherein]

I was asking if the two files will pose any conflicts with the existing code base, or can I just replace those two without making my own version of the compiler, but I see I can't, there is a ClassCastException.

[SethTisue]

Once there's an open PR, you'll be able to test the PR validation snapshot; and once the PR is merged, you'll be able to use the next nightly.