Move GHC.RTS.Flags out of base.

[doyougnu]

What

Presently, we expose GHC.RTS.Flags in base. This proposal proposes to move them out of base and into ghc-internal or some other similar package. We can decide where would be best in the comments.

Why

It is non-obvious that have GHC.RTS.Flags in base creates downstream issues, but in practice this is the case and we end up creating more work for the CLC, for GHC developers, increase coupling of base to GHC versions. There are several CLC proposals which were only created because these flags are exposed in base, and as far as I can tell there isn't much debate in any of the proposals:

• Add --optimistic-linking to GHC RTS Flags exposed in base. #288: adds a RTS linker flag, which becomes exposed in base, and therefore requires a CLC proposal
• Minor tweaks to GHC RTS flags exposed via base #263: created in response to @dcoutts work on the RTS IO managers. This modified the RTS flags in response to the new manager thereby requiring a CLC proposal.
• Add new RTS options introduced by eras profiling to GHC.RTS.Flags  #254: added new RTS flags for profiling modes
• Expose --no-automatic-time-samples in GHC.RTS.Flags API #243: similar to Add new RTS options introduced by eras profiling to GHC.RTS.Flags  #254

Which begs the question: Is there a meaningful debate to be had for RTS flag changes like this? And should that debate occur in base with the CLC. So far these seem to have been relatively uncontroversial.

Implementation

I will implement and this should miss the 9.12 fork. I'm unsure if ghc-internal is the best home for these flags. I'll bring this up in the next GHC call. But I think it is the best place we have right now.

The plan is given by this comment. To be succinct we:

1. create compat functions and types in base. This is MR!13428 and is implemented, waiting on review.
2. move the flags to ghc-experimental and expose them through ghc-experimental. This is where consumers should expect to use these flags moving forward. This is MR!13343, it still needs work but should be trivial I just won't have time until this Friday.
3. Begin deprecating by adding warnings to the compat functions and types from (1). Not yet done.

Impact

As Duncan writes in #263:

The GHC RTS flags are currently exposed via base in GHC.RTS.Flags. The types there reflect 1:1 the GHC RTS CLI flags. Thus these types are obviously tightly coupled to GHC and the GHC version.

This is exactly right, and so decoupling these flags is a meaningful step towards the ghc-base split.

[hasufell]

Does anyone feel this needs a deprecation period?

[mixphix]

I think that because the API is kept in such a close tandem with the compiler that we should be able to get away without a deprecation period, provided that we find a new home for the module where it remains up-to-date with the compiler as current. I suggested ghc-internal in another thread but there may be a more suitable place I'm not aware of. I think in this case telling users to depend on ghc-internal for the compiler flags isn't such an unreasonable request.

[adamgundry]

Thanks for pushing this forward @doyougnu! I agree that RTSFlags should be moved out of base to reduce unnecessary coupling.

Users are strongly discouraged from depending on ghc-internal, so we should expose it somewhere else. To my mind ghc-experimental is the obvious default home for all user-facing but closely-coupled-to-GHC definitions. That said, a new package would also be possible (as @TeofilC suggested in https://gitlab.haskell.org/ghc/ghc/-/issues/25275#note_585663), and could in principle be created with support for existing GHC versions so users could migrate immediately. However it comes with added long-term maintenance burden as it would need to be updated on each GHC release. This is ultimately a decision for the GHC maintainers.

I can see a few plausible migration strategies:

• Simply move GHC.RTS.Flags to a different package in GHC 9.14. This feels like it might break quite a few packages and seems rather aggressive, though the fix is confined to .cabal files.
• Keep GHC.RTS.Flags frozen in base for a few releases (with a deprecation warning); meanwhile add the new module under a different name. This means users have to change their imports to use the new version, but won't be broken as aggressively (or potentially ever, because it seems like the existing API could in principle be maintained indefinitely).
• Re-export the same GHC.RTS.Flags module from both base and the new home for a while, before removing it from base, and encourage users to switch. Unfortunately I don't think reexported-modules can have warnings attached (unlike individual definition re-exports), so it's unclear how effectively this will be at avoiding breakage.

[parsonsmatt]

I feel like ghc-experimental is more for things which are in the "gathering feedback from users" phase - ie, "here's some new API, try it out, it's experimental so we want experience reports, subject to change etc." Whereas ghc-internal is for stuff that's not particularly experimental or controversial, just tightly coupled to GHC implementation details.

Ultimately I think it'd be nice to see GHC.RTS.Flags defined in ghc-internal, and a separate package offering a "compatibility shim" in a backwards/forwards compatible design.

Unfortunately I don't think reexported-modules can have warnings attached (unlike individual definition re-exports), so it's unclear how effectively this will be at avoiding breakage.

This seems like a very useful feature to have on the cabal side for any project that wants to extract modules from one library to another - I'll open an issue.

[tomjaguarpaw]

I agree with @parsonsmatt

[adamgundry]

Adding user-facing definitions to ghc-internal seems to be inconsistent with https://github.com/haskellfoundation/tech-proposals/blob/main/proposals/accepted/051-ghc-base-libraries.rst#proposal, which says:

We should develop both social and technical mechanisms to discourage people from depending directly on ghc-internal, because if such dependencies become frequent and ossified, it will lead to future pain when the API changes.

Moreover it suggests as a candidate for inclusion in ghc-experimental:

future APIs to access RTS statistics, which are fairly stable and user-exposed, but which are (by design) coupled closely to GHC's runtime and hence may change.

Just because a package queries some information about the RTS, I don't think it should be forced to depend on GHC internals.

[tomjaguarpaw]

Adding user-facing definitions to ghc-internal seems to be inconsistent with ...

Just because a package queries some information about the RTS, I don't think it should be forced to depend on GHC internals.

I agree that users should absolutely not get the module in question from ghc-internal. My suggestion would be that they should get it from the "compatibility shim" module than @parsonsmatt hypothesises. Indirect dependencies on ghc-internal should absolutely be encouraged, assuming the author of the intermediate package has reason to believe she/he can keep the API stable. After all, base re-exports many entities from ghc-internal!

If GHC HQ really thinks that ghc-experimental is the correct place then so be it. That's up to them. That said, I'm surprised to see the "GHC's base libraries" proposal suggesting it. It doesn't fit under "experimental" as I understand the concept.

[Ericson2314]

Agreed with @tomjaguarpaw

IIRC the current plans for template-haskell are also a mix of better reinstallability and compatibility shims. This should be thought about in the same way.

[parsonsmatt]

Yeah, I guess for me -experimental asks the question "What is the experiment?" whereas -internal immediately indicates "these are internal details to a specific GHC." The linked definitions don't agree with my intuition on what those names mean (but that's on me as much as anything else)

[parsonsmatt]

Cabal upstream issue mentioned: haskell/cabal#10384

[doyougnu]

I too agree with @parsonsmatt and @Ericson2314 and think ghc-internal is the right home because these flags are by their nature coupled to specific GHC releases. But that does not mean that these definitions are user facing. Instead the user facing definitions should be a shim that I am more than happy to keep in base. The goal here is to be able to alter the RTS flags without opening a CLC proposal and causing a breakage. I feel that the ghc-internal + shim in base approach reaches this goal. Furthermore, I believe that the ghc-base-libraries proposal covers this case:

Over time, the GHC developers may make CLC proposals to remove types and functions that are currently in the base API, but are in truth part of GHC's implementation, and were originally exposed by historical accident. But these are future proposals. ...

This is one such proposal. It continues:

To expose a new function from ghc-internal requires that any functions on which it depends are also in ghc-internal (not base). So we may need to move code from base to ghc-internal, leaving a shim behind in base. In practice, that may mean that quite a lot of code will move into ghc-internal quite quickly. But that's fine: it is just an implementation matter: provided the modules, exports, and API of base are maintained, it is immaterial to clients (and hence to CLC) exactly how they are maintained.

This seems to be exactly the path forward we're discussing here.

Does anyone feel this needs a deprecation period?

I desperately do not want a deprecation period and I believe that a deprecation period would exacerbate the ecosystem coupling. Instead I prefer a clean break with a new GHC release. We are in a fortunate position because we still have a coupled ghc and base. So we can do the whole migration at once for a single release of ghc+base. As for the impact, the same 23 packages that were previously impacted will be impacted again.

[tomjaguarpaw]

I desperately do not want a deprecation period and I believe that a deprecation period would exacerbate the ecosystem coupling.

I don't think there will be any objection to moving the original source of GHC.Flags to a GHC-internal package, re-exporting it from its existing location. However, isn't your ultimate aim to change the API of that module? If so, then I don't think that performing the move helps you. You'll still need CLC approval to make the change.

If, on the other hand, you get CLC approval to remove the module from base (so you can put it wherever you like) then you will be completely free from CLC approval regarding changing that module in future.

[Ericson2314]

I think the idea is that the old version in base should either be removed right away, or frozen in place as a permanent back-compat hack. Newer user-facing versions should live with in a separate library (analogous to template-haskell).