Partial compilation using MIR-only rlibs

[davidtwco]

Proposal

MIR-only rlibs are an idea that has been thrown around for years (rust-lang/rust#38913, rust-lang/rust#48373 and #713 more recently). MIR-only rlibs have always been conceptualised as an artefact that you would use as a dependency to a downstream compilation, compiling anything you need from the MIR-only rlib on-demand. Instead, this MCP proposes "finishing off" a MIR-only rlib to produce a "full" rlib with object files, without needing the original crate source, only the MIR-only rlib.

@wesleywiser and I discussed using a mechanism like this for something like -Zbuild-std: Ship a MIR-only rlib for std, and -Zbuild-std would compile it to a regular rlib to be used as normal. Different compilation flags can be passed to the "finishing off" compilation (mostly codegen flags), enabling std to be built with sanitizers or a different mangling scheme, for example. Cross-compilation could just use the MIR-only rlib from the other target's rustup component. We could even ship MIR-only rlibs for tier three targets that users can "finish off" on their own. It wouldn't be helpful for cfgs, but we don't need to do everything at once. @saethlin suggested it might be useful for miri which just wants all of the MIR for std.

I have a very rough prototype implementation for this on my mir-only-rlib branch. I haven't done too much testing yet, it's capable of finishing-off a simple test case from its MIR-only rlib - it just proves that the idea is technically possible:

• We can already make MIR-only rlibs, use -Zno-codegen -Zalways-encode-mir for this.
• I wanted to be able to make this "just work" as much as possible by taking the query system and instead of using local providers for LOCAL_CRATE, if this is a MIR-only rlib being compiled, use the extern providers.
  • This need some small driver changes to load the rlib from the input path and then put it into the cstore, and then some simple tweaks to the query system.
• After that, just jump straight into monomorphization and codegen.
  • No significant changes were needed - a few uses of the HIR needed to be put behind queries and their results added to the cross-crate metadata, but nothing major.
  • Only thing I noticed but didn't fix was MonoItem::GlobalAsm's use of the HIR, but it doesn't seem too challenging.
• With my branch, use -Zcodegen-only and pass a rlib as an argument. Temporarily, you need to use --crate-name because otherwise it guesses libfoo instead of foo due to the rlib's filename. You also need --crate-type and I forget why, just a implementation limitation.

Ultimately, it's a little fragile in that you can use a query that requires something not in the cross-crate metadata in codegen and not know - we'd want to find a way to make this more robust. I haven't checked whether non-LLVM backends use more queries that aren't backed by the metadata.

Unlike MIR-only rlibs, implementation of this is quite different because it replaces what the compiler does with LOCAL_CRATE pretty significantly, so I wanted to check that the team agreed with this line of experimentation.

Mentors or Reviewers

@wesleywiser (interested stakeholder), @saethlin (previously interested in MIR-only rlibs)

Process

The main points of the Major Change Process are as follows:

• File an issue describing the proposal.
• A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Comments

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

[rustbot]

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

Concerns or objections to the proposal should be discussed on Zulip and formally registered here by adding a comment with the following syntax:

@rustbot concern reason-for-concern 
<description of the concern> 

Concerns can be lifted with:

@rustbot resolve reason-for-concern 

See documentation at https://forge.rust-lang.org

cc @rust-lang/compiler @rust-lang/compiler-contributors

[cjgillot]

This was never seconded, but has been discussed, and been desired for a long time. I'll second it to ensure that's not being blocked needlessly on process.
@rustbot second

[davidtwco]

While I appreciate this being seconded, I'm not sure I think this is the correct path forward anymore - cfgs are much too pervasive for an approach like this to be particularly feasible.