Description
Proposal
Rustc stores a lot of lengths in u32, which reduces the size of data structures and on-disk formats that contain these lengths. But this strategy limits the space of programs that can be compiled. In general, this is a gamble that our user base will benefit more from the performance improvement of picking a small fixed-size type than the implementation limitation this imposes will cause trouble.
We have a few reports of running into the u32 limit for rmeta table offsets and incr comp allocation offsets:
- ICE when include_bytes-ing ~>1GB of data in lib.rs rust#103607
- 'rustc' panicked at 'called
Result::unwrap()on anErrvalue: TryFromIntError(()) rust#112934 - Disk cache ICE with big array rust#76037
- ICE trying to compile very large file contents to wasm32-wasi and maybe x86_64-unknown-linux-gnu rust#95780
- Compiler error encountered when attempting to get the size of a string created from
[21u8; u32::MAX as usize]rust#111613
This is a proposal to find a way to lift the implementation limitation imposed by using u32 in these two places, and thus be able to compile the programs reported in these issues.
Current implementations of these are:
- Use u64 for incr comp allocation offsets rust#113562
- Adapt table sizes to the contents, accommodating u64 rmeta offsets rust#113542
While the goal of this MCP is to support compiling more programs, it is important that the additional space of supported programs not impose a measurable cost on the average user according to our benchmark suite. In that sense, the perf implications of solutions to this MCP should be evaluated on their own, as opposed to as if they are the cost of adding a feature.
Mentors or Reviewers
Process
The main points of the Major Change Process are as follows:
- File an issue describing the proposal.
- A compiler team member or contributor who is knowledgeable in the area can second by writing
@rustbot second.- Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a
-C flag, then full team check-off is required. - Compiler team members can initiate a check-off via
@rfcbot fcp mergeon either the MCP or the PR.
- Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a
- Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.
You can read more about Major Change Proposals on forge.
Comments
This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.