Cannot test with `r3.1` and then `c3.1.0`

[nicolasstucki]

Compiler version

300a8b6

Minimized example

// tests/pos/mytest/A_1_r3.1.scala
class Foo

// tests/pos/mytest/B_2_c3.1.0.scala
class Bar extends Foo

Output

sbt> testCompilation tests/pos/mytest
... 
No files matched "mytest" in test                                   
[info] Test dotty.tools.dotc.CompilationTests.posMacros started
Compilation failed for: 'tests/pos/mytest'                   
[=======================================] completed (1/1, 1 failed, 3s)
...

Expectation

Should pass.

Should also show the error (see #14303).

[nicolasstucki]

I reproduced it manually an got the following error.

> git checkout 300a8b688b9d671341dad58b097367037015b656
> mkdir out2
> sbt "scalac -d out2 -Yrelease 3.1 tests/pos-macros/backwardCompat-3.1/Macro_1_r3.1.scala"
...
> git checkout release-3.1.0
> mkdir out3
> sbt "scalac -classpath out2 -d out3 tests/pos/mytest/B_2_c3.1.0.scala"
...
error while loading Foo,
class file out2/Foo.class is broken, reading aborted with class dotty.tools.tasty.UnpickleException
TASTy signature has wrong version.
 expected: {majorVersion: 28, minorVersion: 1}
 found   : {majorVersion: 28, minorVersion: 2 [unstable release: 1]}

This TASTy file was produced by an unstable release.
To read this TASTy file, your tooling must be at the same version.
The TASTy file was produced by Scala 3.1.2-RC1-bin-SNAPSHOT-nonbootstrapped-git-c3eaadd.
-- [E006] Not Found Error: tests/pos/mytest/B_2_c3.1.0.scala:1:18 --------------
1 |class Bar extends Foo
  |                  ^^^
  |                  Not found: type Foo

I see that the older compiler cannot read this artifact because we generated non-release version of TASTy.

@prolativ does this make sense from the testing framework perspective?

The same example passes on 3.0.x (as in forwardCompat-nestedSumMirror/forwardCompat-refinedGivens/forwardCompat-strictEquals/forwardCompat-unusedImport) because we did not have this check yet. We need to figure out how to overcome this limitation or we won't be able to use these test after 3.0.2.

[prolativ]

@nicolasstucki This looks like a broader question how we understand the correspondence between versions of TASTy and scala. E.g. scala 3.1.0 emits TASTy 28.1-0, so do 3.1.1-RC1, 3.1.1-RC2 and 3.1.1, and so will all nonsnapshot 3.1.x versions. However in our master branch TASTy minor and experimental versions are temporarily bumped to higher values and then downgraded during release. This makes sense when we're preparing a new 3.x.0 release because the TASTy format has not been fixed for this release. But I'm wondering if later snapshot versions like 3.1.2-RC1-SNAPSHOT should produce experimental TASTy instead of the stable version as theoretically they cannt change the format.
Initially 3.1.2-RC1-SNAPSHOT with -Yscala-release 3.1 indeed produced stable 28.1-0 TASTy, similarly as it produced 28.0-0 with -Yscala-release 3.0 but this caused some other problems in tests and was against the intuition that Yscala-release 3.x should work as noop for any 3.x.y* compiler

[prolativ]

As far as I remember there were problems when we tried to compile something like A_1_r3.1.scala with B_2.scala because the latter file was compiled with the latest snapshot compiler from the branch and TASTY 28.1-0 and 28.2-1 are incompatible. In your case it's the other way round - A would produces unstable TASTy 28.2-1 and B uses the compiler using TASTy 28.1-0

[nicolasstucki]

A possible solution would be to not fetch the published artifacts but have a submodule for each supported version containing the release branches (for example https://github.com/lampepfl/dotty/tree/release-3.0.2) and then compile it locally. This would allow us to compile locally a version of the compiler that is equivalent to the released version but would be marked as a snapshot. Therefore it would be able to consume nonstable artifacts.

This would also help in another direction we haven't considered in the past. If a test fails and we need to debug it we might need to debug what the released version of the compiler receives to fully understand the interaction between the two compilers. Therefore if we have a submodule we could modify it (for example to enable the Printers) and run the tests on that version. I have already hit this situation.

[prolativ]

That makes sense. Although all the previously released versions of the compiler that we would like to use in tests should are probably cached in our CI in order not to make the builds last too long

[smarter]

I'm wary of using submodules here, we already have a hard time managing our current usage of submodules without adding even more complexity in the mix. I would definitely prefer an escape hatch for the tests than this.