Skip to content

Add -Zsplit-metadata flag to reduce disk usage #851

Closed
@Kobzol

Description

@Kobzol

Proposal

I would like to propose an unstable flag for rustc tentatively named -Zsplit-metadata. It would instruct the compiler to avoid storing metadata in rlib and dylib artifacts. Instead, it would only store a small metadata stub, and the full metadata would be kept only in the .rmeta file. When using such a compiled artifact as a dependency, the user would have to provide both the rlib/dylib and a corresponding .rmeta file (passed through --extern), which would contain the full metadata contents.

The main motivation for this flag is to reduce compiled artifact disk usage, and potentially also to improve compilation performance slightly when used with Cargo on systems with slow I/O/FS, by writing less data to disk (in exchange for reading a little bit more data, as the compiler would now have to read .rmeta files when linking the final leaf artifact).

Currently, when Cargo compiles a crate, by default it uses --emit=metadata for intermediate dependencies, which tells them to emit a .rmeta file, in addition to the main .rlib file. This allows Cargo to use pipelined compilation, which improves compilation speed. However, it also means that after the compilation, both the .rmeta and .rlib files are stored on the disk, and the crate metadata stored inside of them is thus duplicated. And since the metadata can be quite large, this can waste non-trivial amounts of disk space.

Preliminary results that show how much size could be saved by using this approach can be found here.

This idea has been floated around since a long time ago (rust-lang/rust#23366, rust-lang/rust#29511, rust-lang/rust#57076).

@bjorn3 has created an initial implementation, and under his guidance I cleaned it up a little bit and created an MVP in rust-lang/rust#137535.

If this gets accepted, the plan for landing this would be approximately this:

  1. Land -Zsplit-metadata
  2. Teach Cargo to use it (e.g. behind an unstable Cargo flag)
  3. Dogfood -Zsplit-metadata for the compiler itself in bootstrap
  4. Stabilize the flag

Mentors or Reviewers

@bjorn3, @petrochenkov

Process

The main points of the Major Change Process are as follows:

  • File an issue describing the proposal.
  • A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
    • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
    • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
  • Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Metadata

Metadata

Assignees

No one assigned

    Labels

    T-compilerAdd this label so rfcbot knows to poll the compiler teammajor-changeA proposal to make a major change to rustcmajor-change-acceptedA major change proposal that was accepted

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions