Pre-cache official Ruby binaries on GitHub Actions images instead of custom binaries

[AlenaSviridenko]

We are the maintainers of actions/virtual-environments repo and owns the Hosted images for GitHub Actions and Azure DevOps. We are looking for some collaboration and improving UX of our customers in Ruby area.

Currently, we support two tasks related to Ruby: actions/setup-ruby (for GitHub Actions) and UseRuby (for Azure DevOps). Both tasks require pre-built binaries of Ruby for Linux and MacOS to be placed on images. We build them by ourselves from source code and then put to the images during image generation.

Although this approach works, we believe that a more proper way is to use official Ruby binaries through the https://github.com/ruby/setup-ruby task that is provided by you. You have more expertise in Ruby building and supporting. Actually, ruby/setup-ruby is already mentioned in starter workflow as a default way to use Ruby for GitHub Actions https://github.com/actions/starter-workflows/blob/main/ci/ruby.yml#L27.

The only benefit of actions/setup-ruby is that it understands when Ruby version is already pre-cached on image and doesn't download it again. Action ruby/setup-ruby always downloads specified version.

We think that it would be great if we can collaborate more and pre-cache binaries provided by ruby-builder to our images:

1. You will be able to configure your task to switch Ruby immediately without latency and download in runtime only in case if it is not pre-cached
   • Currently, we pre-cache the latest patch version for every major-minor pair of Ruby on images. It would speed up the task in the most of the cases.
2. We will be able to avoid maintenance of own ruby-builder from our side

We have done investigation in order to investigate differences between ruby-builder and how we build Ruby from our side.
The main difference is that you build with shared libs support that means that binaries are bound to location where they were built and can’t be run from different location.

• You build your binaries under $HOME/.rubies/ruby-<version>.
• We build our binaries under /opt/hostedtoolcache/Ruby/<version>.
  NOTE: hostedtoolcache is recommended approach to store cached tools on images. actions/toolkit works with it by default. Also both actions/setup-ruby (GitHub Actions) and UseRuby (Azure DevOps) tasks are expecting to consume binaries from hostedtoolcache.

Is it possible to change the ruby-builder directory and build binaries under hostedtoolcache? If so, we would be able to achieve better customers UX and get benefits that were mentioned above.

Thank you,
Alyona.

cc: @maxim-lobanov, @alepauly, @sergey-akhalkov

[eregon]

Thanks for opening this issue, that makes sense to me.

So the logic would be like:

• If tc.find('Ruby', version)
  • Use that
• Otherwise use the current logic (download the prebuilt Ruby)

Currently, we pre-cache the latest patch version for every major-minor pair of Ruby on images.

Only Ruby >= 2.4 currently (2.4, 2.5, 2.6, 2.7), right?

Is RUNNER_TOOL_CACHE guaranteed to always be /opt/hostedtoolcache/ on Linux & /Users/runner/hostedtoolcache on macOS?
I guess not for self-hosted runners (not the main subject of this issue, but would be nice to find a Ruby in the hostedtoolcache of a self-hosted runner).
$RUNNER_TOOL_CACHE seems to be currently /opt/hostedtoolcache on Linux, /Users/runner/hostedtoolcache on macOS and C:/hostedtoolcache/windows on Windows.

It might be possible to make the Ruby builds work in any directory. I'm not sure how feasible/reliable is that, but it would be convenient.

Is it OK to write to the hostedtoolcache, so even downloaded versions would be placed there for consistency?

Ruby gems will get installed under the Ruby prefix by default, and I would very much like to avoid messing with GEM_HOME/GEM_PATH variables, as that can lead to various non-trivial issues.
That means, the Ruby prefix must be writable.

What about Windows?
This action uses builds from RubyInstaller. For Ruby >= 2.4 it downloads the .7z archive (without DevKit), since there is already a MSYS2 toolchain on GitHub Actions. URLs are here.
So if the same archives are already extracted in the hostedtoolcache, we should be able to reuse them.

On Windows, we noticed the D: drive seems faster than C:, probably because it seems to be a SSD (#14).
But the hostedtoolcache is on C:.
The initial directory when running an action (pwd) is D:, so that seems the "drive to write to when running a workflow".
So this could lead to a slowdown when installing gems and when installing a Ruby version not in the toolcache.
I'm not sure what's a good way to address that.
We could measure again how much difference it makes.

Alternative Ruby implementations like JRuby & TruffleRuby would always be downloaded, unless you consider adding them to the hostedtoolcache too?

[maxim-lobanov]

So the logic would be like:
Is it OK to write to the hostedtoolcache, so even downloaded versions would be placed there for consistency?

Yes, exactly. If version is not found, download it and do tc.cacheDir. cacheDir will write to hostedtoolcache. It won't be useful on Hosted machines because it is cleaned up after every build but very useful for self-hosted. See full documentation in toolkit.
The good example is setup-java task: https://github.com/actions/setup-java/blob/main/src/installer.ts
Something like that:

let toolPath = tc.find
if (toolPath) {
    core.debug(`Tool found in cache ${toolPath}`);
} else {
  // installation logic
  toolPath = await tc.cacheDir

Is RUNNER_TOOL_CACHE guaranteed to always be /opt/hostedtoolcache/ on Linux & /Users/runner/hostedtoolcache on macOS?

Yes, these paths won't be changed in future because too many things depend on it.

Only Ruby >= 2.4 currently (2.4, 2.5, 2.6, 2.7), right?
Alternative Ruby implementations like JRuby & TruffleRuby would always be downloaded, unless you consider adding them to the hostedtoolcache too?

Yes, we have to find balance between disk space on image and customers' build speed. We don't have telemetry on GitHub Actions but in our understanding, 4 latest versions are most popular and caching them will cover ~90% use-cases. If customers need any specific old version and freeze patch version, they will have minor latency to install in runtime.

As for the Windows

On Windows we use very tricky scheme:
Download exe from RubyInstaller, install it to machine, install MSYS. Then we pack installation directory to archive and unpack this archive to machine during image-generation.
Exe url is https://github.com/oneclick/rubyinstaller2/releases/download/RubyInstaller-${Version}-1/rubyinstaller-${Version}-1-${Architecture}.exe. This code is internal one but you can find result packages that we install on image by this link: https://github.com/orgs/actions/packages?tab=packages&q=ruby
I think we can switch to your approach without any issues on Windows.

It might be possible to make the Ruby builds work in any directory. I'm not sure how feasible/reliable is that, but it would be convenient.

I could be wrong, but in my understanding, if Ruby was built with --enable-shared flag, it will work only in the same directory where it was built.
We have to support binaries for both GitHub Actions task (at least until it is fully deprecated) and Azure DevOps task(it is the main way to use Ruby in Azure DevOps). Both these tasks can't be changed to different directory so it is the reason why we are asking to rebuild your binaries on hostedtoolcache paths to work our of box.

[MSP-Greg]

I think we can switch to your approach without any issues on Windows.

JFYI, all Windows builds used here are self contained, as they contain the dll's needed at runtime, and all the dll's are 'manifested' (their location is linked in the exe), so the 'builds' can be placed anywhere. None use the internal MSYS2 layout.

[MSP-Greg]

@maxim-lobanov

At present, the naming/folder structure for Windows Ruby versions in the toolcache and this action are different. Is that a concern? Most scripts would rely on the Ruby version that 'Path' activates, but there could be scripts that are expecting the current layout?

EDIT:

I knew I had it somewhere in Actions workflows. This action uses rubyinstaller-2.5.8-1-x64, the toolcache uses 2.5.8/x64.

Also, Windows Rubies are currently available in both 64 & 32 bit versions. I'm somewhat active in repos/gems testing & building with Windows, and I don't recall any requests for 32 bit versions. Have you had requests for them in the toolcache?

[maxim-lobanov]

@MSP-Greg , Yes, naming/structure could be concern.
Both GitHub Actions task and Azure DevOps task are expecting Ruby to be placed on default tool-cache structure, so tc.find is able to find it automatically.
Toolcache use the following structure:

- <toolcache_root>
--- ruby
----- 2.5.8
------- x64
--------- <content of Ruby>
----- 2.7.2
------- x64
--------- <content of Ruby>

actions/toolkit contains two methods: tc.find and tc.cacheDir so you don't need to take care about structure.

As I mentioned above, It could be something like:

if (tc.find != null) {
  // use
} else {
  // old logic to download and install on flight
  tc.cacheDir
 // use

As for the 32 bit versions, we have never seen requests for it so I think we should put x64 versions to the image cache and x32 can be downloaded in runtime with latency as currenty.

[MSP-Greg]

x32 can be downloaded in runtime

Sorry I wasn't clearer. At present ruby/setup-ruby does not support 32 bit Windows versions. Or, It won't download them, and there's no way to tag the actions inputs that they're 'requested'.

[maxim-lobanov]

We have never seen requests for 32 bit Ruby on Windows.
I just mentioned that If you would like to add support x32 bit Ruby to this action in future, you will have to add additional input like arch (it could be optional and set as x64 by default but customers can override it to x32). 64 bit Ruby will be switched immediately because it is pre-cached on image. 32 bit will be downloaded.

[eregon]

tc.cacheDir() copies all extracted files, that seems suboptimal as it might take some extra time, and it will create an extra copy on disk (so less free space).

Is there a way to get the toolcache path, so we could extract directly to the right location?

I think it would be better if we always have the Ruby prefix consistently under toolcache, whether or not we had to download, and whether or not it's self-hosted.

Regarding having a build independent of the build directory, I found these configure options:
./configure --enable-shared --enable-rpath --enable-load-relative. I will try that.

[eregon]

I tried building Ruby with those flags, results at https://github.com/ruby/ruby-builder/releases/tag/load-relative

One issue is that gem install bundler fails on --enable-load-relative Rubies:
https://github.com/ruby/setup-ruby/runs/1336222778?check_suite_focus=true

  /home/runner/.rubies/ruby-2.7.0/bin/gem install bundler -v ~> 2 --no-document
  ERROR:  Error installing bundler:
  	"bundle" from bundler conflicts with /home/runner/.rubies/ruby-2.7.0/bin/bundle
  Took   0.56 seconds
Error: The process '/home/runner/.rubies/ruby-2.7.0/bin/gem' failed with exit code 1

Because the existing bin/bundle starts like:

#!/bin/sh
# -*- ruby -*-
_=_\
=begin
bindir="${0%/*}"
exec "$bindir/ruby" "-x" "$0" "$@"
=end
#!/usr/bin/env ruby
#
# This file was generated by RubyGems.
#
# The application 'bundler' is installed as part of a gem, and
# this file is here to facilitate running it.
#

require 'rubygems'

instead of

#!/home/eregon/.rubies/ruby-2.7.2/bin/ruby
#
# This file was generated by RubyGems.
#
# The application 'bundler' is installed as part of a gem, and
# this file is here to facilitate running it.
#

require 'rubygems'

And RubyGems doesn't seem to notice it's safe to override unfortunately.
RubyGems expects the This file was generated by RubyGems line to be the third line:
https://github.com/rubygems/rubygems/blob/6d7fe84753/lib/rubygems/installer.rb#L220

[eregon]

Interestingly gem install rake works without error, and the original bin/rake uses a variant:

#!/bin/sh
# -*- ruby -*-
# This file was generated by RubyGems.
#
# The application 'rake' is installed as part of a gem, and
# this file is here to facilitate running it.
#
_=_\
=begin
bindir="${0%/*}"
exec "$bindir/ruby" "-x" "$0" "$@"
=end
#!/usr/bin/env ruby

require 'rubygems'

rake is a bundled gem and not a default gem, probably that's why it works differently.

[eregon]

Removing the #!/bin/sh part seems to work around this and is easy enough: ruby/ruby-builder@f3e1b0e
https://github.com/ruby/setup-ruby/runs/1336674146?check_suite_focus=true
It should not be done for rake though (can be tested with gem install rake).