Unable to set classpath via @<argument-file> from `dist/bin/scala` command line

[philwalk]

Compiler version

Scala code runner version 3.1.0-RC2 -- Copyright 2002-2021, LAMP/EPFL

Minimized command line

The test script, named importTest.sc:

#!/opt/scala3/bin/scala
import better.files._
def main (args: Array[String]): Unit =
  printf("import test\n")

# verify scala version:
philwalk@d5:/opt/ue$ /opt/scala3/bin/scala -version
Scala code runner version 3.1.0-RC2 -- Copyright 2002-2021, LAMP/EPFL

# verify contents of @<argument-file>:
philwalk@d5:/opt/ue$ cat argumentFile.txt
-classpath /opt/uejlib3/better-files_2.13-3.9.1.jar

# verify ability to set -classpath directly:
philwalk@d5:/opt/ue$ /opt/scala3/bin/scala -classpath /opt/uejlib3/better-files_2.13-3.9.1.jar jsrc/importTest.sc
import test

# demonstrate failure when setting classpath via @<argument-file>:
philwalk@d5:/opt/ue$ /opt/scala3/bin/scala @argumentFile.txt jsrc/importTest.sc

Output

-- [E006] Not Found Error: /opt/ue/jsrc/importTest.sc:2:7 ----------------------------------------------------------------------------------
2 |import better.files._
  |       ^^^^^^
  |       Not found: better

longer explanation available when compiling with `-explain`
1 error found
Error: Errors encountered during compilation
philwalk@d5:/opt/ue$ 

Expectation

Should be able to set the -classpath via @argumentFile.txt on the command line, as with previous releases.

Am able to demonstrate the problem in each of the following bash environments:

Linux quadd 5.4.0-81-generic #91~18.04.1-Ubuntu SMP Fri Jul 23 13:36:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Linux d5 5.4.72-microsoft-standard-WSL2 #1 SMP Wed Oct 28 23:40:43 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
CYGWIN_NT-10.0 d5 3.2.0(0.340/5/3) 2021-03-29 08:42 x86_64 Cygwin
MSYS_NT-10.0-19042 d5 3.2.0-340.x86_64 2021-07-02 08:36 UTC x86_64 Msys

Seems like this would have been caught by compiler/test/dotty/tools/scripting/BashScriptsTests.scala.

[bishabosha]

i noticed that the regexes for scalaOption and colorOption are not capturing anything, but maybe that is not necessary:
https://github.com/lampepfl/dotty/blob/3a6a9499f0bd1821351f052631a0a1d3842f05c1/compiler/src/dotty/tools/MainGenericRunner.scala#L96-L97

[philwalk]

This problem is a side-effect of the order that command line arguments are being passed at MainGenericRunner lines 165-167.
The scalaArgs should be passed before the -script argument. Here's what it should look like:

      case ExecuteMode.Script =>
        val properArgs =
          List("-classpath", settings.classPath.mkString(classpathSeparator)).filter(Function.const(settings.classPath.nonEmpty))
            ++ settings.residualArgs
            ++ (if settings.save then List("-save") else Nil)
            ++ settings.scalaArgs
            ++ List("-script", settings.targetScript)
            ++ settings.scriptArgs
        scripting.Main.main(properArgs.toArray)

This bug also manifests when -color:never is passed, as it gets converted to a script argument.

However, after the above re-ordering of arguments to scripting.Main, there's a residual problem with the classpath. Here's the new error message:

Flag -classpath set repeatedly
Could not find package scala from compiler core libraries.
Make sure the compiler core libraries are on the classpath.

A user could work around this by prepending compiler core libraries to their classpath, although it would seem preferrable to for MainGenericRunner to do it automatically. That's apparently how it's handled in scala3-3.0.2, since it doesn't require the user to add compiler libraries to their classpath, and it doesn't complain about Flag -classpath set repeatedly.

Here's the relevant section of dist/bin/scala lines 178-196:

    [[ $save_compiled == true ]] && rm -f $target_jar
    PROG_NAME=$ScriptingMain
    compilerJavaClasspathArgs # initialize jvm_cp_args with toolchain classpath
    scripting_string="-script $target_script ${script_args[@]}"
    # use eval instead of exec, to insure that onExit is subsequently called

    # $script_cp_arg must be the first argument to $ScriptingMain
    # $scripting_string  must be last
    eval "\"$JAVACMD\"" \
       ${JAVA_OPTS:-$default_java_opts} \
       "${java_args[@]}" \
       "-classpath \"$jvm_cp_args\"" \
       -Dscala.usejavacp=true \
       "$setScriptName" \
       "$ScriptingMain"  \
        ${script_cp_arg-} \
       "${scala_args[@]}" \
       "${residual_args[@]}" \
       "${scripting_string-}"    # must be the last arguments
    scala_exit_status=$?

Perhaps because the compiler toolchain libraries are added to the java classpath, and the @argument-file is a scala argument, there's no conflict, and the 2nd classpath doesn't supercede the 1st?