Unstable compilation output of src/library due to stray references to Predef.class before Predef.scala is named

[retronym]

reproduction steps

using Scala 2.12.12, and the extended version of DeterminismTest I'm working on in retronym/scala#96, the pickle for scala.Symbol.class differs depending on the order/subset of source files being compiled:

sbt:root> junit/test:runMain scala.tools.nsc.DeterminismTester /Users/jz/code/scala/src/library

--- /var/folders/22/g1sv634d11j1d_lqlnhz9p2r0000gn/T/reference1538611564431125389/scala/Symbol.class
+++ /var/folders/22/g1sv634d11j1d_lqlnhz9p2r0000gn/T/recompileOutput2404148374994820649/scala/Symbol.class
@@ -1,17 +1,17 @@
 // class version 52.0 (52)
 // access flags 0x31
 public final class scala/Symbol implements scala/Serializable {

   // compiled from: Symbol.scala

-  @Lscala/reflect/ScalaSignature;(bytes="\u0006\u0001!4Aa\u0004\u0009\u0003'!A1\u0004\u0001BC\u0002\u0013\u0005A\u0004\u0003\u0005)\u0001\u0009\u0005\u0009\u0015!\u0003\u001e\u0011\u0015I\u0003\u0001\"\u0003+\u0011\u0015i\u0003\u0001\"\u0011/\u0011\u0015y\u0003\u0001\"\u00031\u0011\u0015\u0009\u0005\u0001\"\u0011C\u0011\u00151\u0005\u0001\"\u0011H\u000f\u0015i\u0005\u0003#\u0001O\r\u0015y\u0001\u0003#\u0001P\u0011\u0015I\u0013\u0002\"\u0001T\u0011\u0015!\u0016\u0002\"\u0011V\u0011\u00159\u0016\u0002\"\u0005Y\u0011\u0015Q\u0016\u0002\"\u0005\\\u0011\u001dy\u0013\"!A\u0005\n\u0005\u0014aaU=nE>d'\"A\u0009\u0002\u000bM\u001c\u0017\r\\1\u0004\u0001M\u0019\u0001\u0001\u0006\r\u0011\u0005U1R\"\u0001\u0009\n\u0005]\u0001\"AB!osJ+g\r\u0005\u0002\u00163%\u0011!\u0004\u0005\u0002\r'\u0016\u0014\u0018.\u00197ju\u0006\u0014G.Z\u0001\u0005]\u0006lW-F\u0001\u001e!\u0009qRE\u0004\u0002 GA\u0011\u0001\u0005E\u0007\u0002C)\u0011!EE\u0001\u0007yI|w\u000e\u001e \n\u0005\u0011\u0002\u0012A\u0002)sK\u0012,g-\u0003\u0002'O\u000911\u000b\u001e:j]\u001eT!\u0001\n\u0009\u0002\u000b9\u000cW.\u001a\u0011\u0002\rqJg.\u001b;?)\u0009YC\u0006\u0005\u0002\u0016\u0001!)1d\u0001a\u0001;\u0005AAo\\*ue&tw\rF\u0001\u001e\u0003-\u0011X-\u00193SKN|GN^3\u0015\u0003E\u0002\"!\u0006\u001a\n\u0005M\u0002\"aA!os\"\u001aQ!\u000e!\u0011\u0007U1\u0004(\u0003\u00028!\u00091A\u000f\u001b:poN\u0004\"!\u000f \u000e\u0003iR!a\u000f\u001f\u0002\u0005%|'\"A\u001f\u0002\u0009)\u000cg/Y\u0005\u0003�i\u0012Qc\u00142kK\u000e$8\u000b\u001e:fC6,\u0005pY3qi&|gnI\u00019\u0003!A\u0017m\u001d5D_\u0012,G#A\"\u0011\u0005U!\u0015BA#\u0011\u0005\rIe\u000e^\u0001\u0007KF,\u0018\r\\:\u0015\u0005![\u0005CA\u000bJ\u0013\u0009Q\u0005CA\u0004C_>dW-\u00198\u0009\u000b1;\u0001\u0019A\u0019\u0002\u000b=$\u0008.\u001a:\u0002\rMKXNY8m!\u0009)\u0012bE\u0002\n!b\u0001B!F)\u001eW%\u0011!\u000b\u0005\u0002\u0010+:L\u0017/^3oKN\u001c8)Y2iKR\u0009a*A\u0003baBd\u0017\u0010\u0006\u0002,-\")1d\u0003a\u0001;\u0005aa/\u00197vK\u001a\u0013x.\\&fsR\u00111&\u0017\u0005\u000671\u0001\r!H\u0001\rW\u0016LhI]8n-\u0006dW/\u001a\u000b\u00039~\u00032!F/\u001e\u0013\u0009q\u0006C\u0001\u0004PaRLwN\u001c\u0005\u0006A6\u0001\raK\u0001\u0004gflG#\u00012\u0011\u0005\r4W\"\u00013\u000b\u0005\u0015d\u0014\u0001\u00027b]\u001eL!a\u001a3\u0003\r=\u0013'.Z2u\u0001")

Diffing extra trace output added to the pickler:

analysis

We end up with two symbols representing type alias Predef.String = java.lang.String. One of these symbols is created when the Predef.class is loaded from the classpath when loading the scala.package package object

java.lang.Throwable
	at scala.reflect.internal.Types$AliasTypeRef.$init$(Types.scala:1957)
	at scala.reflect.internal.Types$AliasNoArgsTypeRef.<init>(Types.scala:2411)
	at scala.reflect.internal.Types$TypeRef$.apply(Types.scala:2423)
	at scala.reflect.internal.pickling.UnPickler$Scan.readType(UnPickler.scala:415)
	at scala.reflect.internal.pickling.UnPickler$Scan.$anonfun$readTypeRef$1(UnPickler.scala:656)
	at scala.reflect.internal.pickling.UnPickler$Scan.at(UnPickler.scala:190)
	at scala.reflect.internal.pickling.UnPickler$Scan.readTypeRef(UnPickler.scala:656)
	at scala.reflect.internal.pickling.UnPickler$Scan.readNonEmptyTree(UnPickler.scala:605)
	at scala.reflect.internal.pickling.UnPickler$Scan.readTree(UnPickler.scala:619)
	at scala.reflect.internal.pickling.UnPickler$Scan.$anonfun$readAnnotArg$1(UnPickler.scala:467)
	at scala.reflect.internal.pickling.UnPickler$Scan.at(UnPickler.scala:190)
	at scala.reflect.internal.pickling.UnPickler$Scan.readAnnotArg(UnPickler.scala:467)
	at scala.reflect.internal.pickling.UnPickler$Scan.readAnnotationInfo(UnPickler.scala:501)
	at scala.reflect.internal.pickling.UnPickler$Scan.readSymbolAnnotation(UnPickler.scala:513)
	at scala.reflect.internal.pickling.UnPickler$Scan.$anonfun$run$2(UnPickler.scala:108)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.reflect.internal.pickling.UnPickler$Scan.runAtIndex(UnPickler.scala:90)
	at scala.reflect.internal.pickling.UnPickler$Scan.run(UnPickler.scala:108)
	at scala.reflect.internal.pickling.UnPickler.unpickle(UnPickler.scala:47)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.unpickleOrParseInnerClasses(ClassfileParser.scala:1182)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.parseClass(ClassfileParser.scala:466)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.$anonfun$parse$2(ClassfileParser.scala:160)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.$anonfun$parse$2$adapted(ClassfileParser.scala:146)
	at scala.reflect.internal.util.ReusableInstance.using(ReusableInstance.scala:30)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.$anonfun$parse$1(ClassfileParser.scala:146)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.pushBusy(ClassfileParser.scala:129)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.parse(ClassfileParser.scala:145)
	at scala.tools.nsc.symtab.SymbolLoaders$ClassfileLoader.doComplete(SymbolLoaders.scala:333)
	at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:240)
	at scala.reflect.internal.Symbols$Symbol.completeInfo(Symbols.scala:1542)
	at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
	at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:355)
	at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:410)
	at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader.doComplete(SymbolLoaders.scala:303)
	at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:240)
	at scala.reflect.internal.Symbols$Symbol.completeInfo(Symbols.scala:1542)
	at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
	at scala.reflect.internal.Types$TypeRef.decls(Types.scala:2285)
	at scala.tools.nsc.typechecker.Namers$Namer.enterPackage(Namers.scala:765)
	at scala.tools.nsc.typechecker.Namers$Namer.dispatch$1(Namers.scala:288)
	at scala.tools.nsc.typechecker.Namers$Namer.standardEnterSym(Namers.scala:301)
	at scala.tools.nsc.typechecker.AnalyzerPlugins.pluginsEnterSym(AnalyzerPlugins.scala:479)
	at scala.tools.nsc.typechecker.AnalyzerPlugins.pluginsEnterSym$(AnalyzerPlugins.scala:478)
	at scala.tools.nsc.Global$$anon$5.pluginsEnterSym(Global.scala:483)
	at scala.tools.nsc.typechecker.Namers$Namer.enterSym(Namers.scala:279)
	at scala.tools.nsc.typechecker.Analyzer$namerFactory$$anon$1.apply(Analyzer.scala:50)
	at scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:454)
	at scala.tools.nsc.Global$GlobalPhase.run(Global.scala:402)

This is referred to when computing the root imports:

java.lang.Throwable
	at scala.reflect.internal.Symbols$AliasTypeSymbol.<init>(Symbols.scala:3059)
	at scala.reflect.internal.Symbols$Symbol.createAliasTypeSymbol(Symbols.scala:1331)
	at scala.reflect.internal.Symbols$Symbol.newNonClassSymbol(Symbols.scala:1397)
	at scala.reflect.internal.pickling.UnPickler$Scan.readSymbol(UnPickler.scala:346)
	at scala.reflect.internal.pickling.UnPickler$Scan.$anonfun$run$1(UnPickler.scala:98)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.reflect.internal.pickling.UnPickler$Scan.runAtIndex(UnPickler.scala:90)
	at scala.reflect.internal.pickling.UnPickler$Scan.run(UnPickler.scala:98)
	at scala.reflect.internal.pickling.UnPickler.unpickle(UnPickler.scala:47)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.unpickleOrParseInnerClasses(ClassfileParser.scala:1182)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.parseClass(ClassfileParser.scala:466)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.$anonfun$parse$2(ClassfileParser.scala:160)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.$anonfun$parse$2$adapted(ClassfileParser.scala:146)
	at scala.reflect.internal.util.ReusableInstance.using(ReusableInstance.scala:30)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.$anonfun$parse$1(ClassfileParser.scala:146)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.pushBusy(ClassfileParser.scala:129)
	at scala.tools.nsc.symtab.classfile.ClassfileParser.parse(ClassfileParser.scala:145)
	at scala.tools.nsc.symtab.SymbolLoaders$ClassfileLoader.doComplete(SymbolLoaders.scala:333)
	at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:240)
	at scala.reflect.internal.Symbols$Symbol.completeInfo(Symbols.scala:1542)
	at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
	at scala.reflect.internal.Symbols$Symbol.tpeHK(Symbols.scala:1468)
	at scala.reflect.internal.Types$Type.computeMemberType(Types.scala:735)
	at scala.reflect.internal.Types$Type.memberType(Types.scala:732)
	at scala.reflect.internal.TreeGen.mkAttributedSelect(TreeGen.scala:248)
	at scala.reflect.internal.TreeGen.mkAttributedRef(TreeGen.scala:174)
	at scala.reflect.internal.TreeGen.mkAttributedStableRef(TreeGen.scala:209)
	at scala.tools.nsc.ast.TreeGen.mkImportFromSelector(TreeGen.scala:43)
	at scala.tools.nsc.ast.TreeGen.mkWildcardImport(TreeGen.scala:33)
	at scala.tools.nsc.typechecker.Contexts.$anonfun$rootContext$1(Contexts.scala:111)
	at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
	at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
	at scala.collection.immutable.List.foldLeft(List.scala:91)
	at scala.collection.TraversableOnce.$div$colon(TraversableOnce.scala:179)
	at scala.collection.TraversableOnce.$div$colon$(TraversableOnce.scala:179)
	at scala.collection.AbstractTraversable.$div$colon(Traversable.scala:108)
	at scala.tools.nsc.typechecker.Contexts.rootContext(Contexts.scala:111)
	at scala.tools.nsc.typechecker.Contexts.rootContext$(Contexts.scala:110)
	at scala.tools.nsc.Global$$anon$5.rootContext(Global.scala:483)
	at scala.tools.nsc.typechecker.Analyzer$namerFactory$$anon$1.apply(Analyzer.scala:50)
	at scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:454)
	at scala.tools.nsc.Global$GlobalPhase.run(Global.scala:402)
	at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1511)
	at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1495)
	at scala.tools.nsc.Global$Run.compileSources(Global.scala:1488)
	at scala.tools.nsc.DeterminismTester.compile$1(DeterminismTester.scala:56)
	at scala.tools.nsc.DeterminismTester.$anonfun$test$7(DeterminismTester.scala:97)
	at scala.tools.nsc.DeterminismTester.$anonfun$test$7$adapted(DeterminismTester.scala:94)
	at scala.collection.immutable.List.foreach(List.scala:431)
	at scala.tools.nsc.DeterminismTester.test(DeterminismTester.scala:94)
	at scala.tools.nsc.DeterminismTester$.main(DeterminismTester.scala:34)
	at scala.tools.nsc.DeterminismTester.main(DeterminismTester.scala)

Later, the namer enters a source file loader for scala.Predef, running through this code:

def enterClassSymbol(tree: ClassDef): Symbol = {
      val existing = context.scope.lookup(tree.name)
      val isRedefinition = (
           existing.isType
        && existing.isTopLevel
        && context.scope == existing.owner.info.decls
        && currentRun.canRedefine(existing)
      )
      val clazz: Symbol = {
        if (isRedefinition) {
          updatePosFlags(existing, tree.pos, tree.mods.flags)
          setPrivateWithin(tree, existing)
          clearRenamedCaseAccessors(existing)
          existing
        }
        else enterInScope(assignMemberSymbol(tree)) setFlag inConstructorFlag
      }
      clazz match {
        case csym: ClassSymbol if csym.isTopLevel => enterClassSymbol(tree, csym)
        case _                                    => clazz
      }
    }

updatePosFlags resets the existing symbolOf[Predef.type] to represent the source file. But it seems we've already performed some typechecking sees (potentially stale) information from Predef.class. The error mode here isn't related to stateness, but rather to having two distinct identical symbols for the same entity, a detail which shows up in the number of entries of in the pickle.

We should see if this problem generalizes to user code (that isn't redefining Predef from source). Even if the problem is ours alone, a fix is desirable as it would let us use src/library as a regression test for determism without needed to doctor to the test to pass Predef.scala first on the source file list or somesuch.

[som-snytt]

I wonder if 😕 plus ❤️ adequately conveys the feeling, in the absence of 😿 .

[retronym]

Any and all emojis you have to spare are gladly accepted at my Patreon. Platinum level patrons get exclusive access to live-streamed compiler debugging sessions.

[retronym]

Here's a variant in userland, where the package object does not participate in root imports.

  @Test def testPackageObjectUserLand(): Unit = {
    def code = List[SourceFile](
      source("package.scala", "package userland; object `package` { type Throwy = java.lang.Throwable }"),
      source("th.scala", "package userland; class th[T <: Throwy](cause: T = null)")
    )
    test(code :: Nil, permuter = _.reverse :: Nil)
  }

When th.scala appears before package.scala, naming the PackageDef in th.scala with:

    def enterPackage(tree: PackageDef) {
      val sym = createPackageSymbol(tree.pos, tree.pid)
      tree.symbol = sym
      newNamer(context.make(tree, sym.moduleClass, sym.info.decls)) enterSyms tree.stats
    }

... forces the info of the the package class symbol, including entering the members of the package.class. Again, we end up with two symbols representing the same entity which results in a different (albeit semantically equivalent) pickle. I'm pretty sure there is also a correctness bug here in that references to members of package.class that no longer in package.scala might not result in "no such symbol" errors.

[dwijnand]

Pushed as I assumed you wouldn't work on this in the next 10 days...

[retronym]

Possibly helpful is this idea for lazier package object loading: scala/scala#9664