"assertion failed: unresolved symbols: type s2 (line 10) #53305 when pickling" on VirtusLab iskra

[tribbloid]

Compiler version

3.6.2

Minimized code

This is not caused by code but pickling/serialization

Can be reproduced by compiling:

https://github.com/acyclic-ai/six-scala/tree/iskra/dev1 (with 1 git submodule)

using ./gradlew build

Output

## Exception when compiling 30 sources to /home/peng/git/dottyspike/six-scala/module/iskra/build/classes/scala/main
java.lang.AssertionError: assertion failed: unresolved symbols: type s2 (line 10) #53305 when pickling /home/peng/git/dottyspike/six-scala/module/iskra/src/main/scala/org/virtuslab/iskra/Grouping.scala
scala.runtime.Scala3RunTime$.assertFailed(Scala3RunTime.scala:8)
dotty.tools.dotc.core.tasty.TreePickler.pickle(TreePickler.scala:950)
dotty.tools.dotc.transform.Pickler.run$$anonfun$1$$anonfun$1(Pickler.scala:306)
scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
scala.collection.immutable.List.foreach(List.scala:334)
dotty.tools.dotc.transform.Pickler.run$$anonfun$1(Pickler.scala:279)
scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
scala.collection.immutable.List.foreach(List.scala:334)
dotty.tools.dotc.transform.Pickler.run(Pickler.scala:278)
dotty.tools.dotc.core.Phases$Phase.runOn$$anonfun$1(Phases.scala:380)
scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
scala.collection.immutable.List.foreach(List.scala:334)
dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:373)
dotty.tools.dotc.transform.Pickler.runPhase$1(Pickler.scala:392)
dotty.tools.dotc.transform.Pickler.runOn(Pickler.scala:398)
dotty.tools.dotc.Run.runPhases$1$$anonfun$1(Run.scala:343)
scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1323)
dotty.tools.dotc.Run.runPhases$1(Run.scala:336)
dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:384)
dotty.tools.dotc.Run.compileUnits$$anonfun$adapted$1(Run.scala:396)
dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:69)
dotty.tools.dotc.Run.compileUnits(Run.scala:396)
dotty.tools.dotc.Run.compileSources(Run.scala:282)
dotty.tools.dotc.Run.compile(Run.scala:267)
dotty.tools.dotc.Driver.doCompile(Driver.scala:37)
dotty.tools.xsbt.CompilerBridgeDriver.run(CompilerBridgeDriver.java:141)
dotty.tools.xsbt.CompilerBridge.run(CompilerBridge.java:22)
sbt.internal.inc.AnalyzingCompiler.compile(AnalyzingCompiler.scala:91)
sbt.internal.inc.MixedAnalyzingCompiler.$anonfun$compile$7(MixedAnalyzingCompiler.scala:193)
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
sbt.internal.inc.MixedAnalyzingCompiler.timed(MixedAnalyzingCompiler.scala:248)
sbt.internal.inc.MixedAnalyzingCompiler.$anonfun$compile$4(MixedAnalyzingCompiler.scala:183)
sbt.internal.inc.MixedAnalyzingCompiler.$anonfun$compile$4$adapted(MixedAnalyzingCompiler.scala:163)
sbt.internal.inc.JarUtils$.withPreviousJar(JarUtils.scala:239)
sbt.internal.inc.MixedAnalyzingCompiler.compileScala$1(MixedAnalyzingCompiler.scala:163)
sbt.internal.inc.MixedAnalyzingCompiler.compile(MixedAnalyzingCompiler.scala:211)
sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileInternal$1(IncrementalCompilerImpl.scala:534)
sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileInternal$1$adapted(IncrementalCompilerImpl.scala:534)
sbt.internal.inc.Incremental$.$anonfun$apply$5(Incremental.scala:180)
sbt.internal.inc.Incremental$.$anonfun$apply$5$adapted(Incremental.scala:178)
sbt.internal.inc.Incremental$$anon$2.run(Incremental.scala:464)
sbt.internal.inc.IncrementalCommon$CycleState.next(IncrementalCommon.scala:116)
sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:56)
sbt.internal.inc.IncrementalCommon$$anon$1.next(IncrementalCommon.scala:52)
sbt.internal.inc.IncrementalCommon.cycle(IncrementalCommon.scala:263)
sbt.internal.inc.Incremental$.$anonfun$incrementalCompile$8(Incremental.scala:419)
sbt.internal.inc.Incremental$.withClassfileManager(Incremental.scala:506)
sbt.internal.inc.Incremental$.incrementalCompile(Incremental.scala:406)
sbt.internal.inc.Incremental$.apply(Incremental.scala:172)
sbt.internal.inc.IncrementalCompilerImpl.compileInternal(IncrementalCompilerImpl.scala:534)
sbt.internal.inc.IncrementalCompilerImpl.$anonfun$compileIncrementally$1(IncrementalCompilerImpl.scala:488)
sbt.internal.inc.IncrementalCompilerImpl.handleCompilationError(IncrementalCompilerImpl.scala:332)
sbt.internal.inc.IncrementalCompilerImpl.compileIncrementally(IncrementalCompilerImpl.scala:425)
sbt.internal.inc.IncrementalCompilerImpl.compile(IncrementalCompilerImpl.scala:137)
org.gradle.api.internal.tasks.scala.ZincScalaCompiler.execute(ZincScalaCompiler.java:158)
org.gradle.api.internal.tasks.scala.ZincScalaCompilerFacade.execute(ZincScalaCompilerFacade.java:39)
org.gradle.api.internal.tasks.scala.ZincScalaCompilerFacade.execute(ZincScalaCompilerFacade.java:26)
org.gradle.api.internal.tasks.compile.daemon.AbstractIsolatedCompilerWorkerExecutor$CompilerWorkAction.execute(AbstractIsolatedCompilerWorkerExecutor.java:78)
org.gradle.workers.internal.DefaultWorkerServer.execute(DefaultWorkerServer.java:63)
org.gradle.workers.internal.AbstractClassLoaderWorker$1.create(AbstractClassLoaderWorker.java:54)
org.gradle.workers.internal.AbstractClassLoaderWorker$1.create(AbstractClassLoaderWorker.java:48)
org.gradle.internal.classloader.ClassLoaderUtils.executeInClassloader(ClassLoaderUtils.java:100)
org.gradle.workers.internal.AbstractClassLoaderWorker.executeInClassLoader(AbstractClassLoaderWorker.java:48)
org.gradle.workers.internal.IsolatedClassloaderWorker.run(IsolatedClassloaderWorker.java:49)
org.gradle.workers.internal.IsolatedClassloaderWorker.run(IsolatedClassloaderWorker.java:30)
org.gradle.workers.internal.WorkerDaemonServer.run(WorkerDaemonServer.java:103)
org.gradle.workers.internal.WorkerDaemonServer.run(WorkerDaemonServer.java:72)
org.gradle.process.internal.worker.request.WorkerAction$1.call(WorkerAction.java:152)
org.gradle.process.internal.worker.child.WorkerLogEventListener.withWorkerLoggingProtocol(WorkerLogEventListener.java:41)
org.gradle.process.internal.worker.request.WorkerAction.lambda$run$1(WorkerAction.java:149)
org.gradle.internal.operations.CurrentBuildOperationRef.with(CurrentBuildOperationRef.java:85)
org.gradle.process.internal.worker.request.WorkerAction.run(WorkerAction.java:141)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.base/java.lang.reflect.Method.invoke(Method.java:569)
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:48)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.base/java.lang.Thread.run(Thread.java:840)
           

> Task :six:iskra:compileScala FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':six:iskra:compileScala'.
> assertion failed: unresolved symbols: type s2 (line 10) #53305 when pickling /home/peng/git/dottyspike/six-scala/module/iskra/src/main/scala/org/virtuslab/iskra/Grouping.scala

Expectation

it almost contains identical code to the original iskra, only with version changes and an extra compiler argument "-source:3.3", so it should compile successfully, with identical warning

[Gedochao]

cc @prolativ

[prolativ]

@tribbloid what iskra's commit is this based on? As far as I can see it's not the newest one from main (VirtusLab/iskra@8e87d66). Please try updating your code to this commit. It changed some usages of match types which became invalid after scala 3.3.x so that might be the reason of the problem.
Also a notable difference between your code and the original iskra repo is bumping spark to 4.0.0-preview2 although this doesn't seem to cause any problems by itself.

[tribbloid]

@prolativ thanks a lot and great work for inventing it! Yeah I was playing with it by trying to inline the repo in a gradle build (as scala-cli doesn't have a native IntelliJ-IDEA plugin at the moment)

the Scala & Spark version were upgraded to see if there is any performance difference

[tribbloid]

this case clearly need minimisation but so far I haven't found a way to reliably reproduce it.

I heard that bloop/metals has a remote telemetry mode for submitting bug report with deep stack but not sure how it works

[prolativ]

I took an earlier commit from iskra's repo as a starting point and managed to minimize the issue:

//> using scala "3.6.2"
//> using option "-source:3.3"

import scala.quoted.*

type TupleSubtype[T <: Tuple] = T

type AsTuple[T] = T match
    case TupleSubtype[t] => t
    case _ => EmptyTuple

trait Foo:
  type Tpe

object GroupedDataFrame:
  def bar(foo: Expr[Foo])(using Quotes) =
    foo match
      case '{ $f: Foo { type Tpe = t } } =>
        List.empty[AsTuple[t]]

It looks like a regression as it didn't use to crash the compiler in 3.5.0

[prolativ]

Bisection seems to point to d421f88
@EugeneFlesselle