start publishing Scala snapshots again

[SethTisue]

so that people can specify their Scala version as e.g. 2.11.8-SNAPSHOT and 2.12.0-SNAPSHOT and it just works.

this was formerly handled on the old CI at e.g. https://scala-webapps.epfl.ch/jenkins/view/2.11.x/job/scala-nightly-main-2.11.x/ but has not been moved to the new CI

we want to publish them as snapshots, available from e.g.https://oss.sonatype.org/content/repositories/snapshots/org/scala-lang/scala-library/2.11.8-SNAPSHOT/ and https://oss.sonatype.org/content/repositories/snapshots/org/scala-lang/scala-library/2.12.0-SNAPSHOT/

we are already publishing distributions nightly to e.g. http://www.scala-lang.org/files/archive/nightly/2.11.x/ but they are distribution packages only, not usable via sbt's set scalaVersion := ...

reported/requested by @dragos. he writes:

there’s three use cases that come to mind:

• reproducing a bug on “master" before reporting it
• integration builds with “fragile” projects. Besides IDE (which is quite tightly coupled to the compiler), I think scala-async or scala-meta might do this.
• benefit from a bug fix in the presentation compiler (like the one Jason just fixed). There are no more milestones on 2.11.8 (edited)

but the issue is I have no idea what git hash I want, I just want the latest one. It’s an integration build. I want to see if anything broke since yesterday

Anyway, the reason nobody else said anything might also be because this wasn’t discussed or announced anywhere, internally or externally. People might simply not know (2.11.8-SNAPSHOT still resolves to the Sept. 15 version)

[SethTisue]

@lrytz comments:

just need to find the right spot in https://github.com/scala/scala/blob/2.11.x/scripts/jobs/validate/publish-core

[SethTisue]

the question arose whether we actually prefer "nightly" builds or "mergely" builds ("mergely" meaning "whenever a PR is merged")

[dragos]

There's no point in mergely builds. Maven for sure, and Sbt probably, won't check more often than once a day for SNAPSHOT updates.

You could either designate one mergely build as the nightly that gets published as a snapshot, or build and publish once a day, off master (whatever branch that is).

[SethTisue]

We discussed this at the Scala team meeting today and agreed that we (meaning me) will get the snapshots going again just like before. I'm likely to tackle it sooner rather than later just to get it over with.

[SethTisue]

@smarter wants to do this for Dotty as well

[SethTisue]

re mergely vs. nightly, we average 1 or 2 merges/day, so there's no major difference in cost. @retronym says "end users don’t care” so "just do whatever’s easier"

[SethTisue]

fwiw (it's perhaps useful to know that the causes had nothing to do with e.g. how we talk to Sonatype)

• 2.11 nightlies on old Jenkins stopped in September because we merged Abort Ant build if Ant version is <1.9 scala#4746 which requires Ant 1.9, and I guess we had older Ant on that machine. (log of first failure)
• 2.12 nightlies stopped in May because we merged Require and target Java 8 scala#4526 which requires Java 8 but Jenkins was still trying to use Java 6 (log of first failure)

(Adriaan later disabled both jobs)

[SethTisue]

our existing nightly job is integrate/bootstrap (script contents, Jenkins job). the job description claims "This job is run on every merge" but it's clear from the run log that it's actually just running nightly. I don't see in anything our current setup that runs mergely. (I guess the lack of that is #9.)

tweaking integrate/bootstrap to do this looks to me like it will be easier than messing with validate/publish-core. note that integrate/bootstrap is also how we publish real Scala releases, so we know that it has the right Sonatype smarts. whereas publish-core looks to me like it's oriented around publishing to Artifactory (https://scala-ci.typesafe.com/artifactory), not Sonatype.

so, it looks like nightly snapshots it will be.

[SethTisue]

some investigation of the existing integrate/bootstrap script:

• it already has a concept of nightly builds and is already using version numbers of the form 2.12.0-<sha>-nightly but has publishToSonatype=no. so those are the version numbers we see in the distributions that get uploaded to http://www.scala-lang.org/files/archive/nightly/2.11.x/
• otoh, an example of a manually triggered run with publishToSonatype=yes is https://scala-ci.typesafe.com/job/scala-2.11.x-integrate-bootstrap/205/, where SCALA_VER_BASE was manually specified to Jenkins as 2.11.7.)
• note that publishSonatype() publishes both Scala itself and all its modules, too (since the nightly cannot be presumed to be binary compatible with any other Scala version, including any other nightly). in the case of Scala, the actual publishing is done by calling ant publish-signed; in the case of the modules, by with sbt's publish task (after overriding some necessary settings so the right credentials are used and the destination repo is correct).

[SethTisue]

the version numbers of the old snapshots published from the old Jenkins looked like e.g. 2.11.8-20150915-122802-bb52b24bab (this is what I got when I did ;resolvers += Resolver.sonatypeRepo("snapshots") ;scalaVersion := "2.11.8-SNAPSHOT" just now).

is the SNAPSHOT stuff picky about the exact format of our version numbers? do I need to match the old format, or is it OK to stick with the 2.11.8-72b855f-nightly style format? (it's kind of nice to have the date right there in the number, alongside the SHA)

[SethTisue]

integrate/bootstrap is pretty much just calling ant publish-signed — there is a great deal of logic in build.xml that does the rest. for example there are lines like:

<if><contains string="${maven.version.number}" substring="-SNAPSHOT"/><then>
      <property name="remote.repository" value="${remote.snapshot.repository}"/>

so it looks like if we set the variables up right, Ant will take care of publishing to https://oss.sonatype.org/content/repositories/snapshots and not https://oss.sonatype.org/content/repositories/releases