`extra-packages` need to go in `shell-for`

[michaelpj]

Otherwise you get a shell where cabal is unhappy: #1861 (comment)

[hamishmack]

@andreabedini are these included in the plan.json?

[andreabedini]

(Ouch, I had missed this ticket 🫣)

Yes, extra-packages should be in plan.json as they are part of the plan like a local package. Not entirely sure how to identify them as "extra" but maybe we should not have that problem.

[andreabedini]

In #1861 I end up saying

shellFor and cabal repl need to agree re what packages to have in scope.

It turns out they do agree, extra-packages are not in scope in cabal repl.

❯ cat cabal.project
-- hello depends on aeson
packages: ./hello.cabal
extra-packages: lens
❯ cabal exec -- sh -c 'cat $GHC_ENVIRONMENT | grep "\(aeson\|lens\)"'
package-id aeson-2.2.0.0-dcd0d2b64774f31f4ee1a9c03fbf6eafaf340534276728e158adeceb8fad54f0

aeson is in scope as a dependency of hello but lens is not.

Therefore, if the mantra is to do what cabal does, haskell.nix does the right thing here.

[cumber]

@andreabedini I'm not sure checking the environment cabal provides is an accurate test. cabal exec --help says:

Compiler tools will be configured to see the same subset of the store that
builds would see

And I assume that subset is what $GHC_ENVIRONMENT is showing.

The extra-packages aren't dependencies of any build target (and that's the whole point), so they're not going to be in "the same subset of the store that builds would see". They're just supposed to be installed, which makes them accessible to bare ghci (and usable in cabal repl by using :set package commands).

[andreabedini]

@cumber You are right. My view of what shellFor should do is a little superficial.

I had missed some facts about how cabal exec works (see below) and forgotten that cabal exec also adds any executable in the plan to $PATH (check by adding e.g. cabal-plan as an extra-package).

I am happy to take back my statement that "shellFor and cabal repl need to agree re what packages to have in scope" but then my question becomes, if it's not cabal exec or cabal repl, what should use to model shellFor behaviour?

Should we just say that shellFor provides a environment with all project packages and executables in scope? As shown below cabal exec kinda tries to do this but not very reliably.

Some experiments with cabal exec

❯ cabal exec -- sh -c 'cat $GHC_ENVIRONMENT' |  grep package-db
clear-package-db
global-package-db
package-db /home/andrea/.local/state/cabal/store/ghc-9.4.6/package.db
package-db /home/andrea/Scratchpad/haskell-nix-issue-1861/dist-newstyle/packagedb/ghc-9.4.6

notice the entire cabal store is passed in the environment.

Therefore, the following works

❯ cabal exec -- ghci
Loaded package environment from /home/andrea/Scratchpad/haskell-nix-issue-1861/dist-newstyle/tmp/environment.-7203/.ghc.environment.x86_64-linux-9.4.6
GHCi, version 9.4.6: https://www.haskell.org/ghc/  :? for help
Loaded GHCi configuration from /home/andrea/.ghci
λ> :set -package lens
package flags have changed, resetting and loading new packages...

picking up the lens that I have in the cabal store

❯ ghc-pkg --package-db=/home/andrea/.local/state/cabal/store/ghc-9.4.6/package.db list lens
/home/andrea/.local/state/cabal/store/ghc-9.4.6/package.db
    lens-5.2.2

This seems a bit incidental and fragile to me because nothing is guaranteeing that there is at least one lens in the cabal store is not guarateed (and who knows what happens if there are more than one! edit: using -package will let ghc guess while using -pacakge-id you can specify which one you want; tl;dr: -package lens does not guaratee you get the one in the plan).

Starting for a clean slate (no store and no dist-newstyle), cabal exec does not trigger any building and just fails to find the store packagedb.

❯ cabal exec -- ghci -package lens
Resolving dependencies...
Loaded package environment from /home/andrea/Scratchpad/haskell-nix-issue-1861/dist-newstyle/tmp/environment.-11124/.ghc.environment.x86_64-linux-9.4.6
GHCi, version 9.4.6: https://www.haskell.org/ghc/  :? for help
ghc-9.4.6: can't find a package database at /home/andrea/.local/state/cabal/store/ghc-9.4.6/package.db

Doing cabal build alone won't build the extra-packages, you need to build them separately.

❯ cabal build
Build profile: -w ghc-9.4.6 -O1
In order, the following will be built (use -v for more details):
...
❯ cabal exec -- ghci -package lens
Loaded package environment from /home/andrea/Scratchpad/haskell-nix-issue-1861/dist-newstyle/tmp/environment.-19906/.ghc.environment.x86_64-linux-9.4.6
GHCi, version 9.4.6: https://www.haskell.org/ghc/  :? for help
<command line>: cannot satisfy -package lens
    (use -v for more information)
❯ cabal build lens
Build profile: -w ghc-9.4.6 -O1
In order, the following will be built (use -v for more details):
...
Completed    lens-5.2.2 (lib)
❯ cabal exec -- ghci -package lens
Loaded package environment from /home/andrea/Scratchpad/haskell-nix-issue-1861/dist-newstyle/tmp/environment.-22911/.ghc.environment.x86_64-linux-9.4.6
GHCi, version 9.4.6: https://www.haskell.org/ghc/  :? for help
Loaded GHCi configuration from /home/andrea/.ghci
λ>

[cumber]

@andreabedini Yeah, you seem to be right. Cabal doesn't seem to do anything to make extra-packages installed, it just picks them up if they are in the store. I suppose that's deliberate, because the whole point is that building the project shouldn't depend on the extra-packages (otherwise they'd just be dependencies). Arguably cabal repl should? I might file an issue with them and see what they say.

It does actually do solving with them though, they're not simply ignored. If you are in a folder with a cabal.project containing extra-packages: raw-strings-qq and then you manually cabal build raw-strings-qq, it installs version 1.1. Whereas if your cabal.project lists extra-packages: raw-strings-qq ^>= 1.0, then it installs version 1.0.2.

[cumber]

Maybe this is a bump that can't fully be reconciled.

My mental model of how Haskell.Nix is supposed to work is that it sets up a "pre-installed" cabal store in the nix store so that cabal would never need to build/install any other packages. So I had expected that extra-packages would simply be included in that set.

But of course, if I were releasing a finished product build based on Haskell.Nix, I then wouldn't want those extra-packages to be dependencies of the final builds. Only in the dev shell. But if the dev shell and the final builds are produced using the same nix derivation for the cabal store, that can't work. In that case using the nix side additional to control whether they're actually installed or not seems reasonable.

(Actually my initial instinct was to try to add in extra packages to the build plan from the nix side, and was told "As ever, the Haskell.Nix Way is to configure what you want with cabal")

It is unpleasant that using extra-packages alone completely breaks the REPL with little explanation, which wouldn't happen when working with cabal directly. (It does happen when you have a package-db but don't have an index from hackage at all, but that's an extremely unusual state when working with cabal directly). I don't know if anything can be done about that; it seems that Haskell.Nix would either need to filter the extra-packages out of the build plan entirely, or include them in the built store, or have mechanisms for explicitly deciding which of those two you want (additional provides the explicit include; is there anything for the other option?). Or else just say that cabal projects using extra-packages aren't fully supported.

Ideally I would want extra-packages to just magically be available in shellFor environment, but not when the package is finally built (e.g. building an executable output with nix, or including the library as a dependency of some other Haskell project). I don't know whether that's possible though.

Either way, thanks for your time looking into it!

[andreabedini]

@cumber
Let me be clear: I needed to understand your expectations and cabal's behaviour but I didn't mean to argue that your expectations were wrong :)

It does actually do solving with them though, they're not simply ignored. If you are in a folder with a cabal.project containing extra-packages: raw-strings-qq and then you manually cabal build raw-strings-qq, it installs version 1.1. Whereas if your cabal.project lists extra-packages: raw-strings-qq ^>= 1.0, then it installs version 1.0.2.

Yes, extra-packages are part of the plan (with their own constraints). They don't seem to be included in the default build targets though (even cabal build all does not build them).

Ideally I would want extra-packages to just magically be available in shellFor environment, but not when the package is finally built (e.g. building an executable output with nix, or including the library as a dependency of some other Haskell project). I don't know whether that's possible though.

TBH I think this is fair expectation. Once we understand that we are not modelling shellFor after either cabal exec or cabal repl we are free to decide what we want shellFor to be :-).

I believe it should not be too hard to have extra-packages pre-built in the devshell (and only in the devshell). Note that their presence in cabal.project will influence the plan though, as it does with cabal.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.