source-repository-package not optional

[nomeata]

I was hoping that I can use source-repository-package to point to “optional” dependencies, i.e. dependencies that are used when needed, but not built when not. I want to use this in my cabal.project:

source-repository-package
      type:     git
      location: https://github.com/ghcjs/ghcjs-base
      tag: 01014ade3f8f5ae677df192d7c2a208bd795b96c

because ghcjs-base is not on Hackage yet. The problem is that this package is only a dependency of jsaddle when using ghcjs, but not when using ghc. Furthemore, ghcjs-base has dependencies that cannot be fulfilled with ghc.

So with this in cabal.project, cabal thinks it has to build ghcjs-base, even with using ghc, and fails.

[typedrat]

So basically, they shouldn't be added like they're another package in the same project, but they should just expand the universe of things that the solver knows about to depend on?

[nomeata]

At least that was my (possibly naive) expectation. The main reason I want to use this feature is to be able to build packages that are not being uploaded to Hackage, like reflex (reflex-frp/reflex-dom#140, reflex-frp/reflex#177)

[typedrat]

That feels like how it should work to me, too.

[hvr]

This is a general problem; we only have packages: and optional-packages: available; which force packages to become local ones which are solved for (even if they're not necessarily built). So source-repository-packages are basically like having Git submodules + packages:. From that POV, that's consistent.

Otoh, we can add additional package repo indices via repository-stanzas; these would just "expand the universe" w/o them being included always in the build-plan.

What would be useful to have is a way to merely add single packages to the "universe" (similiar to repository-stanzas; but much simpler), and then have a property-field for source-repository-packages in order to declare whether this package ought to be local (like it is now), non-local (i.e. cached in the nix-style store), or index (i.e. merely "added to the universe"; as if it was included in a package repo index).

PS: @nomeata for now you can workaround this by having separate cabal.project files for ghc and ghcjs; you can then use --project-file= to select the "personality" you need.

[gbaz]

That idea (adding packages to the universe) was basically the cabal-side design of the provenance-qualified package imports proposal: ghc-proposals/ghc-proposals#115

Cabal could adopt that or something similar even without GHC actually implementing the ability to track provenance in the packagedb proper.

[nomeata]

The documented use case for `optional-packages`` is vendored dependencies. I guess usually you use your dependencies, so usually people don’t notice the difference, but I would argue that the current behaviour is not following the principle of least surprise.

PS: @nomeata for now you can workaround this by having separate cabal.project files for ghc and ghcjs; you can then use --project-file= to select the "personality" you need.

Doing that already :-)

[hvr]

@nomeata well, the main use-case for source-repository-package as I see it is to cause a specific snapshot of a package to be used in your install-plan; so if you point your cabal.project file to some git repo, I'd argue the principle of least surprise would be in fact require that the Git version is taken, and not some other version that might have been on Hackage.

If we now wanted the variant that such (possibly temporarily) vendored dependencies are declared "exclusive" (i.e. they hide all other versions for the same package-name), but at the same time act like index-packages (i.e. they're only included in the install-plan if anything force them in by actually depending on them), we need yet another mode (strawman: index-exclusive) than the one I suggested in #5444 (comment)

(I'm not sure how provenance-qualification would benefit here; once remote git repo deps are involved, we're not in a self-describing package context anymore, but instead in a cabal.project-specific context where the cabal.project file sets up the "routing" of where packages are procured from -- i.e. that's where the "local" of "nix-style local builds" becomes apparent: we're dealing with local versions of packages which are supposed to supplant/shadow any pre-existing Hackage packages of the same name; I don't think we intend to introduce new names or labelled namespaces when vendor a package in this context -- for example, if I locally vendor a Git snapshot of text, I want that version to be used everywhere across my install-plan, and w/o having to modify any build-depends; nor do I think I care about the provenance information if the pkg is registered in a nix-style pkg store/cache or even merely the local dist-newstyle pkg-db as I don't see what this information would be useful for)

[nomeata]

I'd argue the principle of least surprise would be in fact require that the Git version is taken, and not some other version that might have been on Hackage.

Good point…

index-exlusive sounds right. Not sure if we need all the others :-)

[hvr]

well, you need the local one at least to mark your "primary" local packages; otherwise cabal new-build all would not build anything

PS: we also have extra-packages: which I didn't bring up yet...

and here's a link to the docs for convenience: http://cabal.readthedocs.io/en/latest/nix-local-build.html#specifying-the-local-packages

[nomeata]

The docs don't help me to understand why I'd want extra-packages.

[gbaz]

@hvr, you write

I'm not sure how provenance-qualification would benefit here; once remote git repo deps are involved, we're not in a self-describing package context anymore, but instead in a cabal.project-specific context

But the provenance quantification proposal described how to specify git repo deps as repositories, which is exactly what you're describing. And it described how to do so within cabal files rather than just project files.

Cabal files are extended in two ways.

source-origin-alises: is a top-level property consisting of a list of aliases in the usual indented assignment style. These aliases will be passed to all invocations of ghc in the course of a cabal build. Example syntax:

source-origin-aliases:
hackage: https://…
foobar: http://…

build-depends: fields now can take package names in the form of [REPO]+somepackage rather than just somepackage.

All that is required is to allow source-origin-aliases to take aliases to git repos of single packages as well as standard repositories, which can be derived directly from the syntax we already have for source-repository-package.

[hvr]

(I briefly talked with @gbaz about the provenance qualification; there's overlap in what you can do via cabal.project already now and what prov-qualification would allow, but the intentions of the two mechanisms diverge)

@nomeata that's true; the documentation of extra-packages is very lacking; you can read however Duncan's rationale for it at e.g. #3737 (comment) and a recent enhancement to its syntax (which also hasn't found its way into the docs) was merged via #4889

[nomeata]

This discussion prompted me to not use source-repository-package for my goal, but rather set up my own hackage overlay repository with these packages:
http://hackage-ghcjs-overlay.nomeata.de/
https://github.com/nomeata/hackage-ghcjs-overlay
A bit manual, but makes the travis scripts that uses them easier.