Reordering the compilation classpath causes the compiler to crash

Compiler version

3.1.2

Minimized code

Clone the repository from https://github.com/vasilmkd-jetbrains/zio2-protoquill-example.

Resolve dependencies

Open the cloned repository using sbt, so it can download all of the dependencies.

Reproduction

Using sbt, print the classpath

Open sbt and run export fullClasspath. Copy the value.

Navigate to src/main/scala/example/module

Execute scalac (Scalac 3.1.2):

scalac -cp <the classpath that you copied from 2 steps above> Example.scala Main.scala

Compilation should succeed.

Now for the compiler crash.

Lexicographically sort the classpath

In the Scala repl, copy/paste the classpath.

val classpath = "<classpath value copied from sbt>"
val sorted = classpath.split(":").sorted.mkString(":")
println(sorted)

Again, copy the obtained value.

Execute scalac (Scalac 3.1.2) again:

scalac -cp <the lexicographically sorted classpath from above> Example.scala Main.scala

Result:
Compilation fails and the compiler crashes with the following stack trace:

-- Error: Example.scala:34:9 ---------------------------------------------------
34 |     run(query[Person]).provide(env)
   |         ^^^^^^^^^^^^^
   |undefined: io.getquill.Quoted.apply # -1: TermRef(TermRef(TermRef(ThisType(TypeRef(NoPrefix,module class io)),object getquill),Quoted),apply) at inlining
   | This location contains code that was inlined from ExprModel.scala:222
   | This location contains code that was inlined from GetQuill.scala:3
-- Error: Example.scala:34:9 ---------------------------------------------------
34 |     run(query[Person]).provide(env)
   |         ^^^^^^^^^^^^^
   |undefined: io.getquill.Quoted.apply # -1: TermRef(TermRef(TermRef(ThisType(TypeRef(NoPrefix,module class io)),object getquill),Quoted),apply) at inlining
   | This location contains code that was inlined from QuoteMacro.scala:80
   | This location contains code that was inlined from GetQuill.scala:3
exception occurred while compiling Example.scala, Main.scala
scala.MatchError: Hole(true,2,List()) (of class dotty.tools.dotc.ast.Trees$Hole) while compiling Example.scala, Main.scala
Exception in thread "main" scala.MatchError: Hole(true,2,List()) (of class dotty.tools.dotc.ast.Trees$Hole)
        at dotty.tools.dotc.ast.desugar$.apply(Desugar.scala:1733)
        at dotty.tools.dotc.typer.Typer.typedUnnamed$1(Typer.scala:2856)
        at dotty.tools.dotc.typer.Typer.typedUnadapted(Typer.scala:2872)
        at dotty.tools.dotc.typer.ReTyper.typedUnadapted(ReTyper.scala:121)
        at dotty.tools.dotc.typer.Inliner$InlineTyper.typedUnadapted(Inliner.scala:1686)
        at dotty.tools.dotc.typer.Typer.typed(Typer.scala:2937)
        at dotty.tools.dotc.typer.Typer.typed(Typer.scala:2941)
        at dotty.tools.dotc.typer.Typer.typedInlined(Typer.scala:1850)
        at dotty.tools.dotc.typer.Typer.typedUnnamed$1(Typer.scala:2831)
        at dotty.tools.dotc.typer.Typer.typedUnadapted(Typer.scala:2872)
        at dotty.tools.dotc.typer.ReTyper.typedUnadapted(ReTyper.scala:121)
        at dotty.tools.dotc.typer.Inliner$InlineTyper.typedUnadapted(Inliner.scala:1686)
        at dotty.tools.dotc.typer.Typer.typed(Typer.scala:2937)
        at dotty.tools.dotc.typer.Typer.typed(Typer.scala:2941)
        at dotty.tools.dotc.typer.ProtoTypes$FunProto.$anonfun$3$$anonfun$1(ProtoTypes.scala:416)
        at dotty.tools.dotc.typer.ProtoTypes$FunProto.cacheTypedArg(ProtoTypes.scala:386)
        at dotty.tools.dotc.typer.ProtoTypes$FunProto.$anonfun$3(ProtoTypes.scala:416)
        at dotty.tools.dotc.typer.ProtoTypes$FunProto.$anonfun$adapted$1(ProtoTypes.scala:416)
        at dotty.tools.dotc.core.Decorators$ListDecorator$.loop$3(Decorators.scala:179)
        at dotty.tools.dotc.core.Decorators$ListDecorator$.mapWithIndexConserve$extension(Decorators.scala:186)
        at dotty.tools.dotc.typer.ProtoTypes$FunProto.typedArgs(ProtoTypes.scala:416)
        at dotty.tools.dotc.typer.Applications.realApply$1(Applications.scala:925)
        at dotty.tools.dotc.typer.Applications.typedApply(Applications.scala:1027)
        at dotty.tools.dotc.typer.Applications.typedApply$(Applications.scala:317)
        at dotty.tools.dotc.typer.Inliner$InlineTyper.typedApply(Inliner.scala:1599)
        at dotty.tools.dotc.typer.Typer.typedUnnamed$1(Typer.scala:2809)
        at dotty.tools.dotc.typer.Typer.typedUnadapted(Typer.scala:2872)
        at dotty.tools.dotc.typer.ReTyper.typedUnadapted(ReTyper.scala:121)
        at dotty.tools.dotc.typer.Inliner$InlineTyper.typedUnadapted(Inliner.scala:1686)
        at dotty.tools.dotc.typer.Typer.typed(Typer.scala:2937)
        at dotty.tools.dotc.typer.Typer.typed(Typer.scala:2941)
        at dotty.tools.dotc.typer.Typer.typedInlined(Typer.scala:1850)
        at dotty.tools.dotc.typer.Typer.typedUnnamed$1(Typer.scala:2831)
        at dotty.tools.dotc.typer.Typer.typedUnadapted(Typer.scala:2872)
        at dotty.tools.dotc.typer.ReTyper.typedUnadapted(ReTyper.scala:121)
        at dotty.tools.dotc.typer.Inliner$InlineTyper.typedUnadapted(Inliner.scala:1686)
        at dotty.tools.dotc.typer.Typer.typed(Typer.scala:2937)
        at dotty.tools.dotc.typer.Typer.typed(Typer.scala:2934)
        at dotty.tools.dotc.typer.Typer.typed(Typer.scala:2941)
        at dotty.tools.dotc.typer.Typer.typedExpr(Typer.scala:3057)
        at dotty.tools.dotc.typer.Inliner$InlineTyper.typedApply(Inliner.scala:1604)
        at dotty.tools.dotc.typer.Typer.typedUnnamed$1(Typer.scala:2809)
        at dotty.tools.dotc.typer.Typer.typedUnadapted(Typer.scala:2872)
        at dotty.tools.dotc.typer.ReTyper.typedUnadapted(ReTyper.scala:121)
        at dotty.tools.dotc.typer.Inliner$InlineTyper.typedUnadapted(Inliner.scala:1686)
        at dotty.tools.dotc.typer.Typer.typed(Typer.scala:2937)
        at dotty.tools.dotc.typer.Typer.typed(Typer.scala:2941)
        at dotty.tools.dotc.typer.ReTyper.typedTyped(ReTyper.scala:62)
        at dotty.tools.dotc.typer.Typer.typedUnnamed$1(Typer.scala:2814)
        at dotty.tools.dotc.typer.Typer.typedUnadapted(Typer.scala:2872)
        at dotty.tools.dotc.typer.ReTyper.typedUnadapted(ReTyper.scala:121)
        at dotty.tools.dotc.typer.Inliner$InlineTyper.typedUnadapted(Inliner.scala:1686)
        at dotty.tools.dotc.typer.Typer.typed(Typer.scala:2937)
        at dotty.tools.dotc.typer.Typer.typed(Typer.scala:2934)
        at dotty.tools.dotc.typer.Typer.typed(Typer.scala:2941)
        at dotty.tools.dotc.typer.Inliner.inlined(Inliner.scala:1054)
        at dotty.tools.dotc.typer.Inliner$.inlineCall(Inliner.scala:160)
        at dotty.tools.dotc.typer.Inliner$InlineTyper.inlineIfNeeded(Inliner.scala:1682)
        at dotty.tools.dotc.typer.Inliner$InlineTyper.typedApply(Inliner.scala:1606)
        at dotty.tools.dotc.typer.Typer.typedUnnamed$1(Typer.scala:2809)
        at dotty.tools.dotc.typer.Typer.typedUnadapted(Typer.scala:2872)
        at dotty.tools.dotc.typer.ReTyper.typedUnadapted(ReTyper.scala:121)
        at dotty.tools.dotc.typer.Inliner$InlineTyper.typedUnadapted(Inliner.scala:1686)
        at dotty.tools.dotc.typer.Typer.typed(Typer.scala:2937)
        at dotty.tools.dotc.typer.Typer.typed(Typer.scala:2934)
        at dotty.tools.dotc.typer.Typer.typed(Typer.scala:2941)
        at dotty.tools.dotc.typer.Inliner.inlined(Inliner.scala:1054)
        at dotty.tools.dotc.typer.Inliner$.inlineCall(Inliner.scala:160)
        at dotty.tools.dotc.transform.Inlining$InliningTreeMap.transform(Inlining.scala:86)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform$$anonfun$1(Trees.scala:1493)
        at scala.collection.immutable.List.mapConserve(List.scala:472)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1493)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1389)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:120)
        at dotty.tools.dotc.transform.Inlining$InliningTreeMap.transform(Inlining.scala:84)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1383)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:120)
        at dotty.tools.dotc.transform.Inlining$InliningTreeMap.transform(Inlining.scala:94)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1391)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:120)
        at dotty.tools.dotc.transform.Inlining$InliningTreeMap.transform(Inlining.scala:94)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1389)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:120)
        at dotty.tools.dotc.transform.Inlining$InliningTreeMap.transform(Inlining.scala:84)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:103)
        at dotty.tools.dotc.transform.Inlining$InliningTreeMap.transform(Inlining.scala:80)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.traverse$1(TreeMapWithImplicits.scala:53)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transformStats(TreeMapWithImplicits.scala:60)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:35)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.traverse$1(TreeMapWithImplicits.scala:54)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transformStats(TreeMapWithImplicits.scala:60)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:111)
        at dotty.tools.dotc.transform.Inlining$InliningTreeMap.transform(Inlining.scala:80)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1466)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:120)
        at dotty.tools.dotc.transform.Inlining$InliningTreeMap.transform(Inlining.scala:80)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.traverse$1(TreeMapWithImplicits.scala:53)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transformStats(TreeMapWithImplicits.scala:60)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:35)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.traverse$1(TreeMapWithImplicits.scala:54)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transformStats(TreeMapWithImplicits.scala:60)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:35)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.traverse$1(TreeMapWithImplicits.scala:54)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transformStats(TreeMapWithImplicits.scala:60)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:35)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.recur$1(TreeMapWithImplicits.scala:41)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.traverse$1(TreeMapWithImplicits.scala:54)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transformStats(TreeMapWithImplicits.scala:60)
        at dotty.tools.dotc.ast.Trees$Instance$TreeMap.transform(Trees.scala:1477)
        at dotty.tools.dotc.ast.TreeMapWithImplicits.transform(TreeMapWithImplicits.scala:120)
        at dotty.tools.dotc.transform.Inlining$InliningTreeMap.transform(Inlining.scala:94)
        at dotty.tools.dotc.transform.Inlining$$anon$2.transform(Inlining.scala:72)
        at dotty.tools.dotc.transform.MacroTransform.run(MacroTransform.scala:21)
        at dotty.tools.dotc.transform.Inlining.run(Inlining.scala:44)
        at dotty.tools.dotc.core.Phases$Phase.runOn$$anonfun$1(Phases.scala:308)
        at scala.collection.immutable.List.map(List.scala:246)
        at dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:309)
        at dotty.tools.dotc.transform.Inlining.runOn(Inlining.scala:48)
        at dotty.tools.dotc.Run.runPhases$1$$anonfun$1(Run.scala:259)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
        at scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
        at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1328)
        at dotty.tools.dotc.Run.runPhases$1(Run.scala:270)
        at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:278)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
        at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:68)
        at dotty.tools.dotc.Run.compileUnits(Run.scala:287)
        at dotty.tools.dotc.Run.compileSources(Run.scala:220)
        at dotty.tools.dotc.Run.compile(Run.scala:204)
        at dotty.tools.dotc.Driver.doCompile(Driver.scala:39)
        at dotty.tools.dotc.Driver.process(Driver.scala:199)
        at dotty.tools.dotc.Driver.process(Driver.scala:167)
        at dotty.tools.dotc.Driver.process(Driver.scala:179)
        at dotty.tools.dotc.Driver.main(Driver.scala:209)
        at dotty.tools.dotc.Main.main(Main.scala)

Discussion:

This issue resulted in the IntelliJ IDEA build system, which for reasons that are still not fully understood (I'm working on it), lexicographically sorts the classpath before compilation, which is how we discovered this issue. The related YouTrack issue is here (not fully up to date) https://youtrack.jetbrains.com/issue/SCL-20078/Scala-3.1.1-fails-to-compile-no-method-options.

Is this classpath sorting in IntelliJ an inherent problem and should be changed to follow the classpath reported by sbt, or is this all just an unfortunate set of circumstances? Does the ordering of the classpath given to Scalac matter? I know that the ordering of the classpath matters for execution, and the loading of classes at runtime depends on it, but does the ordering of the classpath matter for Scalac compilation, when provided as an argument using -cp.

Any feedback is greatly appreciated, I'm also working on fixing the issue on the IntelliJ side. It's not my intention to just report this error and expect a fix. Thanks in advance.

[smarter]

does the ordering of the classpath matter for Scalac compilation, when provided as an argument using -cp.

Like at runtime, it should only make a difference if the same class appears in multiple entries of the classpath, if this isn't the case here then this is likely a bug.

[vasilmkd]

I cannot completely rule out whether there are or aren't duplicate classes in this particular example. There are definitely no duplicate jars.

Is there a way I could check for duplicate classes? Thanks.

[smarter]

unzip all the jars and check for duplicates? Depending on your unzipper, maybe you can try unzipping in the same locaiton and emitting an error if a file is overwritten

[smarter]

To make this easier to reproduce you could try giving instructions using coursier. cs fetch "org:name:1.0" will give a full classpath, so you could pipe that into sort and join the lines.

[vasilmkd]

I guess that's cs fetch "io.getquill:quill-jdbc-zio:3.17.0.Beta3.0-RC2" "org.postgresql:postgresql:42.3.4". Sorry if this is wrong, I haven't used standalone Coursier yet.

[vasilmkd]

I will search for duplicate classes in the repo later in the day.

[smarter]

I can reproduce with:

% git clone https://github.com/vasilmkd-jetbrains/zio2-protoquill-example
% cd zio2-protoquill-example
% scalac  -classpath "$(cs fetch "io.getquill:quill-jdbc-zio_3:3.17.0.Beta3.0-RC2" "org.postgresql:postgresql:42.3.4"|sort|perl -pe 'chomp if eof'|tr '\n' ':')" src/main/scala/example/module/Example.scala src/main/scala/example/module/Main.scala

If I replace sort by sort -r to reverse the list, then it compiles.