build-tool-depends executables don't appear in shellFor

[ocharles]

I want to use tasty-discover, which I have added to my .cabal file as:

  build-tool-depends: tasty-discover:tasty-discover

As described in input-output-hk/nix-tools#95, this adds a build-tools entry in the cabal-to-nix expression, which is used in setup-builder.nix to form executableToolDepends. However, it doesn't propagate correctly into the shell produced with shellFor. For build-tool-depends to work, we need a tasty-discover entry in the ghc-pkg database that also has the tasty-discover executable. As things currently stand, there is no entry at all. We can use additional = pkgs: [ pkgs.tasty-discover ], but that will only add a tasty-discover to the package database as a library - no executables will get included.

[michaelpj]

Related/dupe: #760 #231

[ocharles]

Ah, thanks!

[rvl]

I think I know how to fix this now, thanks @ocharles.

[ocharles]

@rvl Oh, fantastic! Do you need any help with anything? Happy to test anything, if that helps!

[JBetz]

A similar issue seems to happen if your shell dependencies have build tool dependencies.

E.g., cardano-transactions has a test suite with a build tool dependency on cardano-tx, and when I try to create a shell on a cabal project that uses it, it errors with:

error: Neither the Haskell package set or the Nixpkgs package set contain the package: cardano-tx (build tool dependency).
If this is a system dependency:
You may need to augment the system package mapping in haskell.nix so that it can be found.
If this is a Haskell dependency:
If you are using Stackage, make sure that you are using a snapshot that contains the package. Otherwise you may need to update the Hackage snapshot you are using, usually by updating haskell.nix.

Not sure if the same thing happens on stack projects, but I would presume not considering that cardano-transactions uses stack.

[michaelpj]

I spent a little bit of time looking at this.

Firstly, I noticed that we don't pass build-tools into the fake component that we use to create the wrapped GHC for shellFor. Easy enough to fix.

But then it still doesn't appear in the package db. It turns out that makeConfigFiles doesn't use build-tools at all. So... how does anything work? My guess is that the Setup.hs interface to cabal doesn't insist on build-tool-depends being in the package db, and so we get away with just providing the executable. But probably we should register them, I'm just not sure how right now.

[Ericson2314]

Yeah I don't think they should be registered in the same package db, especially if package DBs are to be per machine for cross. When you need a library, you usually need lots of information beyond the machine code (unfoldings for inlining, etc.) but for an executable dependency, one just needs to be able to run the thing, and being agnostic to how the sausage is made is probably better for sake of determinism / caching.

[michaelpj]

Yeah I don't think they should be registered in the same package db, especially if package DBs are to be per machine for cross.

Well, not having it in the package db confuses cabal, so I don't see how we can avoid it (if we want this usecase to work!).

As far as cross goes, obviously the build-tools should be in the native package db in the glorious future once we've got a separate native package db. Then I think there's no real problem with them being there.

[matthewbauer]

Note that this really isn't specific shellFor, but specific to v2 cabal. You can always use v1-* cabal commands to get the old behavior (see SolveExecutables in cabal code). I think a reason this is coming up recently is that cabal 3.4.0.0 made v2 the default.

The reason this is really difficult is that both Nixpkgs & haskell.nix use the Setup.hs method of configure / install, which doesn't install a .conf file for executable components. I'm not sure if it makes more sense to patch Cabal so Setup.hs method installs executables correctly, or try to switch to the new install method.

See Cabal comment for reference:

build-tools dependencies
NB: Only include these dependencies if SolveExecutables
is True. It might be false in the legacy solver
codepath, in which case there won't be any record of
an executable we need.

https://github.com/haskell/cabal/blob/master/cabal-install-solver/src/Distribution/Solver/Modular/IndexConversion.hs#L348-L357

[Ericson2314]

Thanks for digging into that!