Calling `quoted.Expr.{run,show}` crashes at runtime in Dotty 0.13.0-RC1, but not 0.12.0

[fhackett]

When testing if #5997 was reproducible on the latest version of Dotty, I found instead that trying to run my example on version 0.13.0-RC1 crashes at runtime. The same code running on 0.12.0 or 0.12.0-RC1 does not crash.

Steps to reproduce (on Ubuntu 16.04, with sbt 1.2.7):

$ git clone git@github.com:lampepfl/dotty-example-project.git
$ cd dotty-example-project
$ git log
commit a753b14e281bbaa6c69f26298913ad6feba969c7
Merge: 3169229 34953cd
Author: Aggelos Biboudis <aggelos.biboudis@epfl.ch>
Date:   Tue Feb 26 19:34:33 2019 +0100

    Merge pull request #25 from lampepfl/dotty-0.13
    
    Release Dotty 0.13.0-RC1
[rest of log omitted]
$ sbt
sbt:dotty-example-project> about
[info] This is sbt 1.2.7
[info] The current project is ProjectRef(uri("file:/.../dotty-example-project/"), "root") 0.1.0
[info] The current project is built against Scala 0.13.0-RC1
[info] Available Plugins
[info]  - dotty.tools.sbtplugin.DottyIDEPlugin
[info]  - dotty.tools.sbtplugin.DottyPlugin
[info]  - sbt.ScriptedPlugin
[info]  - sbt.plugins.CorePlugin
[info]  - sbt.plugins.Giter8TemplatePlugin
[info]  - sbt.plugins.IvyPlugin
[info]  - sbt.plugins.JUnitXmlReportPlugin
[info]  - sbt.plugins.JvmPlugin
[info]  - sbt.plugins.SbtPlugin
[info] sbt, sbt plugins, and build definitions are using Scala 2.12.7
sbt:dotty-example-project> console
scala> import quoted.Toolbox.Default._
scala> val v = '{ (if true then Some(1) else None).map(v => v+1) }
scala> v.show
scala.quoted.Toolbox$ToolboxNotFoundException: Could not load the Toolbox class `dotty.tools.dotc.quoted.ToolboxImpl` from the JVM classpath. Make sure that the compiler is on the JVM classpath.
	at scala.quoted.Toolbox$.make(Toolbox.scala:29)
	at scala.quoted.Toolbox$Default$.make(Toolbox.scala:15)
	at rs$line$4$.<init>(rs$line$4:1)
[huge stacktrace omitted]
Caused by: java.lang.ClassNotFoundException: dotty.tools.dotc.quoted.ToolboxImpl
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at scala.quoted.Toolbox$.make(Toolbox.scala:21)
	... 54 more
scala> v.run
[the same or very similar error]

Let me know if you need any other information in order to reproduce the issue. I abridged the stacktraces for readability, but perhaps I cut out something important.

At first glance, it seems like either the Dotty sbt plugin is misbehaving by somehow not setting the classpath properly, or Dotty is doing something very weird. I don't know enough about the internals to determine which.

[biboudis]

As a side note again, it works on the repl that is on master.

[fhackett]

@biboudis thanks for the note.

I'm curious however what I'm supposed to do with this information as someone writing a project that uses Dotty as a dependency. So far I've been using an sbt project based on the example code linked on the website, which as far as I can tell can only pull release builds.

I understand that using a compiler under heavy development like Dotty is asking for all sorts of issues, and I am prepared to deal with them as they come.

In this context, how would you suggest I set up my project going forward? Do I just rely on the latest release that works for me (that is, 0.12.0 due to this issue), or should I go find some way of tracking master? What would be the most helpful to the Dotty developers: issues with the latest release, with the master branch, or something else?

[biboudis]

Tracking master for fun and possibly contributing is the proper way.

However, if you just want to explore the new compiler in your own projects and just report back (as you did 👍) the binaries should be enough. Consequently, the dotty-example-project should work correctly.

It seems that the runtime dependency of the compiler itself is missing from the sbt configuration. This is because you used the Toolbox. Its implementation resides in dotty.tools.dotc.quoted. If you were running the command dotc, it would be sufficient to pass the -with-compiler flag.

We are currently investigating the best way to make this work within the console of the dotty-example-project.

[smarter]

What would be the most helpful to the Dotty developers: issues with the latest release, with the master branch, or something else?

You can use whatever version works for you, but when reporting a bug, please check if it's still broken on the latest nightly build: https://github.com/lampepfl/dotty-example-project#nightly-builds, since we're not doing point releases currently (unless a release is really broken in some way).

[fhackett]

Thanks for the ideas - I'll probably be moving toward tracking nightly builds (thanks for the link!) so any issues I run into are as relevant as possible. Let's see how this goes :)

[fhackett]

Hello again. I've managed to get a usable workflow using a local checkout of Dotty's master branch and the dotc/dotr scripts, but since I eventually want to have proper build scripting / dependency management for my project I have been looking into getting sbt to work somehow (even hackily).

After some examination of what the dotr script actually does, I have determined that the -with-compiler option effectively adds the following libraries to the classpath:

libraryDependencies ++= Seq(
    "ch.epfl.lamp" %% "dotty-compiler" % latestDotty,
    "ch.epfl.lamp" % "dotty-interfaces" % latestDotty,
    "org.scala-lang.modules" % "scala-asm" % "6.0.0-scala-1",
)

Now from the syntax above you can imagine what I actually attempted. I assumed that maybe if I registered those packages as my project's managed dependencies it would have the same effect, causing them to be present on the classpath, resolving the error that started this issue.

When I tried this nothing happened (as in, dependencies loaded, error persists with no noticeable change), and I have no idea why. Have I misunderstood how sbt works? I count myself as an sbt beginner, so feel free to point me to a relevant part of the manual if it's something obvious.

[biboudis]

Nop, you haven't misunderstood and you are absolutely right. You are doing it correctly. There is something wrong going on with the classloaders between the dotty plugin for sbt and the requirements for quoting. We will report the progress/fix at #6017.

[fhackett]

@biboudis Thanks for the sanity check - good to hear my intuition made sense. I'll keep an eye on that PR for updates.