Support installing cross compilers (such as new JS backend)

[hasufell]

@bgamari @hsyl20 @david-christiansen

Afaiu from talking to Ben, the current temporary approach would be to ship additional bindists specific to cross compiler.

There seems to be work going on to allow to fold that into the same bindist and have a target flag or so for GHC?

I'm not sure what our strategy should be to support this.

[hsyl20]

There seems to be work going on to allow to fold that into the same bindist and have a target flag or so for GHC?
I'm not sure what our strategy should be to support this.

This will likely require support for all boot libraries to be downloaded and installed from Hackage. Also Cabal wouldn't consider them not reinstallable anymore.

After this, the strategy for ghcup will be to only consider the host platform of the compiler, not the target platform. So just like as it does today. So I'm not sure if we should invest too much into ghcup distributing cross host/target bindists (x86_64/js, aarch64/js...) or if we should better wait for multi-target GHC. The latter seems easier to maintain and better for users.

[hasufell]

Well, ghcup can build cross compilers today and actually knows how to distinguish by target platform.

But it's not a widely used feature.

Installing via bindists would also need somewhat more work, but it won't require a major refactoring.

[hsyl20]

Well, ghcup can build cross compilers today and actually knows how to distinguish by target platform.

I didn't know this. So can I already build GHC with the JS backend directly from ghcup? That would be already better than the build instructions we provide (especially if cabal is made to work automatically with this selected ghc)

[hasufell]

I didn't know this. So can I already build GHC with the JS backend directly from ghcup?

Good question. I've only tried building ARM cross compiler (there used to be a test even, which I should probably revive: https://gitlab.haskell.org/haskell/ghcup-hs/-/blob/master/.gitlab-ci.yml#L355 ).

GHCup allows to use both make and hadrian and make is still the default. I have never tried to build a cross compiler via hadrian. I don't know whether it works the same way.

Here's the hadrian specific code:

ghcup-hs/lib/GHCup/GHC.hs

Lines 994 to 1036 in 278a300

	compileHadrianBindist :: ( MonadReader env m
	, HasDirs env
	, HasSettings env
	, HasPlatformReq env
	, MonadThrow m
	, MonadCatch m
	, HasLog env
	, MonadIO m
	, MonadFail m
	)
	=> GHCTargetVersion
	-> FilePath
	-> InstallDirResolved
	-> Excepts
	'[ FileDoesNotExistError
	, HadrianNotFound
	, InvalidBuildConfig
	, PatchFailed
	, ProcessError
	, NotFoundInPATH
	, CopyError]
	m
	(Maybe FilePath) -- ^ output path of bindist, None for cross
	compileHadrianBindist tver workdir ghcdir = do
	lEM $ execWithGhcEnv "python3" ["./boot"] (Just workdir) "ghc-bootstrap"
	liftE $ configureBindist tver workdir ghcdir
	lift $ logInfo "Building (this may take a while)..."
	hadrian_build <- liftE $ findHadrianFile workdir
	lEM $ execWithGhcEnv hadrian_build
	( maybe [] (\j -> ["-j" <> show j] ) jobs
	++ maybe [] (\bf -> ["--flavour=" <> bf]) buildFlavour
	++ ["binary-dist"]
	)
	(Just workdir) "ghc-make"
	[tar] <- liftIO $ findFiles
	(workdir </> "_build" </> "bindist")
	(makeRegexOpts compExtended
	execBlank
	([s|^ghc-.*\.tar\..*$|] :: ByteString)
	)
	liftE $ fmap Just $ copyBindist tver tar (workdir </> "_build" </> "bindist")

The target triple tver is only used in configure:

ghcup-hs/lib/GHCup/GHC.hs

Lines 1187 to 1240 in 278a300

	configureBindist :: ( MonadReader env m
	, HasDirs env
	, HasSettings env
	, HasPlatformReq env
	, MonadThrow m
	, MonadCatch m
	, HasLog env
	, MonadIO m
	, MonadFail m
	)
	=> GHCTargetVersion
	-> FilePath
	-> InstallDirResolved
	-> Excepts
	'[ FileDoesNotExistError
	, InvalidBuildConfig
	, PatchFailed
	, ProcessError
	, NotFoundInPATH
	, CopyError
	]
	m
	()
	configureBindist tver workdir (fromInstallDir -> ghcdir) = do
	lift $ logInfo [s|configuring build|]
	if | _tvVersion tver >= [vver|8.8.0|] -> do
	lEM $ execWithGhcEnv
	"sh"
	("./configure" : maybe mempty
	(\x -> ["--target=" <> T.unpack x])
	(_tvTarget tver)
	++ ["--prefix=" <> ghcdir]
	++ (if isWindows then ["--enable-tarballs-autodownload"] else [])
	++ fmap T.unpack aargs
	)
	(Just workdir)
	"ghc-conf"
	| otherwise -> do
	lEM $ execLogged
	"sh"
	( [ "./configure", "--with-ghc=" <> either id id bghc
	]
	++ maybe mempty
	(\x -> ["--target=" <> T.unpack x])
	(_tvTarget tver)
	++ ["--prefix=" <> ghcdir]
	++ (if isWindows then ["--enable-tarballs-autodownload"] else [])
	++ fmap T.unpack aargs
	)
	(Just workdir)
	"ghc-conf"
	Nothing
	pure ()

And in fact the CLI code will throw an error assuming it doesn't work yet when combining cross with hadrian:

ghcup-hs/app/ghcup/GHCup/OptParse/Compile.hs

Lines 557 to 559 in 278a300

	(CompileGHC GHCCompileOptions { hadrian = True, crossTarget = Just _ }) -> do
	runLogger $ logError "Hadrian cross compile support is not yet implemented!"
	pure $ ExitFailure 9

So if you know what to pass to hadrian to make this work, the changes to ghcup code should be minimal.

[hsyl20]

Hadrian doesn't require specific parameters to build a cross-compiler. Only configure needs to be passed the right toolchain (via CC, LD, etc. The JS backend relies on emconfigure to pass appropriate env variables for a JS target) and --target=....

For the JS backend we have the additional constraint that we need to enable the native backend of ghc-bignum, which is done using the native_bignum flavour transformer of Hadrian.

[hasufell]

Then we could just remove the optparse restriction and try something like:

ghup compile ghc --version=9.6.1 --bootstrap-ghc=9.6.1 --cross-target=javascript-unknown-ghcjs --hadrian --flavour=default+native_bignum

I can only try next week. I'm travelling and don't have a laptop. If you want to give it a shot yourself, I'm happy to assist.

[hsyl20]

I'll be busy this week getting TH merged before the 9.8 fork, but after that sure!

[hsyl20]

I'll give it a shot

[hsyl20]

Thanks a lot @hasufell!