Relativiaze pickled source paths based on rootsource

[nicolasstucki]

We should relativize all source paths to the sourceroot as we did in #10385.
We are missing this in PositionPickler.scala#L63 where we always use the cwd.

[sjrd]

What's sourceroot exactly? In general, source files often come from several "root" source directories. For example src/main/scala and src/main/scala-3.

[liufengyun]

What's sourceroot exactly? In general, source files often come from several "root" source directories. For example src/main/scala and src/main/scala-3.

From the code, the default value is the current working directory of the compilation process:

  val sourceroot: Setting[String] = PathSetting("-sourceroot", "Specify workspace root directory.", ".")

[sjrd]

In that case, it's not very useful as a base to relativize source file names. We should have a different option for that. In sbt it should be filled in by default with the contents of (TheConfig / sourceDirectories).value.

[liufengyun]

In that case, it's not very useful as a base to relativize source file names.

IIUC, currently the main motivation to relativize the path is to address the concern about privacy.

When the tasty is read, no assumptions can be made about the path: the reference point (sourceroot) is unknown, and even it's known, the source may not exist.

It seems to me that having one reference point (instead of multiple reference points) makes potential tools that take advantage of the relative paths easier. @sjrd Could you elaborate a little more about your point below?

In that case, it's not very useful as a base to relativize source file names. We should have a different option for that. In sbt it should be filled in by default with the contents of (TheConfig / sourceDirectories).value.

[sjrd]

When you consume TASTy from somewhere else, you're most likely consuming it from a .jar. That .jar can have an associated source jar. And sources will be stored in the stored jar as relativized to the entries in sourceDirectories. Therefore, if you also relativize the paths in TASTy wrt. the entries in sourceDirectories, tools will be able to reliably relate to the sources in the source jars.

[liufengyun]

When you consume TASTy from somewhere else, you're most likely consuming it from a .jar. That .jar can have an associated source jar. And sources will be stored in the stored jar as relativized to the entries in sourceDirectories. Therefore, if you also relativize the paths in TASTy wrt. the entries in sourceDirectories, tools will be able to reliably relate to the sources in the source jars.

Thanks for the explanation. There seems to be one inconvenience with multiple reference points:

src/root1/util.scala
src/root2/util.scala

If we have both src/root1 and src/root2 as reference points, we will lose the ability to distinguish the two files.

[sjrd]

Yes. You shouldn't do that anyway, because they will erase each other in source jars.

[liufengyun]

Yes. You shouldn't do that anyway, because they will erase each other in source jars.

For the compiler, it has no concept of source jars. That means for the compiler to handle it properly, it has to check the multiple reference points for potential conflicts. That seems to complicate the compiler unnecessarily.

[liufengyun]

A bit of more context about -sourceroot, it's added in 4a3071f to generate relative paths for SemanticDB usage in IDE.

I'm still digging how scala2 handles relative path.

[liufengyun]

In a private conversation, @sjrd explained that the multiple reference points scheme in Scala source jars dates back to java source jars.

In Scala2, it stores only the file name, and the tools seem to have no problem with source jars. If that is the case, we don't have to worry too much about the tools, as with the single reference point scheme, it contains more information to correlate the path in source jars than in Scala 2. This way, we can avoid pushing the complexity to the compiler.

/cc: @romanowski @tgodzik @bishabosha Are you aware of any special needs that the single reference point scheme fails to satisfy but are not a problem in Scala2? Thanks.

[romanowski]

Scala3doc defines yet another similar setting called -sourcepath that is used map source to its location within the repository.

-doc-source-url <url>                       A URL pattern used to link to the source file, with some variables supported: 
For example, for `scala.collection.Seq` €{TPL_NAME} gives `Seq`, €{TPL_OWNER} gives `scala.collection`, €{FILE_PATH} gives `scala/collection/Seq`, €{FILE_EXT} gives `.scala`, €{FILE_PATH_EXT} gives `scala/collection/Seq.scala`, and €{FILE_LINE} gives `25` (without the backquotes). 
To obtain a relative path for €{FILE_PATH} and €{FILE_PATH_EXT} instead of an absolute one, use the -sourcepath setting.

This leaves us with 2 slightly different settings one that will provide relative paths within source jar (-sourceroot) and another to provide relative path within repository (`-sourcepath', doctool specific).

if we aim to use single reference point scheme it should have a semantic similar to -sourcepath otherwise representing a project with e.g. generated sources may be really problematic (often target directory in sbt is not placed withn baseDirectory for given sbt project and e.g. generated sources are placed there)

[liufengyun]

Thanks @romanowski .

if we aim to use single reference point scheme it should have a semantic similar to -sourcepath otherwise representing a project with e.g. generated sources may be really problematic (often target directory in sbt is not placed withn baseDirectory for given sbt project and e.g. generated sources are placed there)

Currently, the default value for the Dotty compiler option -sourceroot is relative to the repository root (.):

• It aligns with the semantics of -sourcepath used in the doc tool
• It aligns with the usage of relative path in semanticDB (4a3071f )

Indeed, it helps a lot for tools that add links to source files in git repositories a lot.