Rust binary (after objcopy) too large

[crepererum]

Abstract

The binary that is flashed via the rust example is over 250MB large, too large for the target system. The C example does NOT suffer from this issue.

Reproduction

1. Get Rust version 1.45.0
2. cd r-riscv-blink
3. cargo build --release (although the issue also exists w/o the release flag)

Technical Details

$ cd target/riscv32i-unknown-none-elf/release/
$ ls -lh r-riscv-blink{,.bin}
-rwxr-xr-x 2 foo bar 456K Jul 25 16:08 r-riscv-blink
-rwxr-xr-x 1 foo bar 257M Jul 25 16:08 r-riscv-blink.bin

So while the input ELF (r-riscv-blink) has a somewhat expected size for an unstripped Rust program, the output binary (r-riscv-blink.bin) is just huge. If we look at the ELF file:

$ readelf -S r-riscv-blink
There are 23 section headers, starting at offset 0x71c18:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .text.dummy       PROGBITS        20000000 0000d4 040000 00   A  0   0  1
  [ 2] .text             PROGBITS        20040000 041000 000a14 00  AX  0   0  4
  [ 3] .rodata           PROGBITS        20040a14 041a14 000054 00   A  0   0  4
  [ 4] .data             PROGBITS        10000000 042000 000000 00   A  0   0  4
  [ 5] .bss              NOBITS          10000000 042000 000004 00  WA  0   0  1
  [ 6] .heap             PROGBITS        10000004 042000 000000 00  WA  0   0  1
  [ 7] .stack            PROGBITS        10000004 042000 01fffc 00  WA  0   0  1
  [ 8] .debug_line       PROGBITS        00000000 061ffc 0017e2 00      0   0  1
  [ 9] .debug_info       PROGBITS        00000000 0637de 00380b 00      0   0  1
  [10] .debug_abbrev     PROGBITS        00000000 066fe9 00051e 00      0   0  1
  [11] .debug_aranges    PROGBITS        00000000 067508 000030 00      0   0  8
  [12] .debug_str        PROGBITS        00000000 067538 003c21 01  MS  0   0  1
  [13] .debug_ranges     PROGBITS        00000000 06b160 0005f0 00      0   0  8
  [14] .riscv.attributes RISCV_ATTRIBUTE 00000000 06b750 00001a 00      0   0  1
  [15] .debug_loc        PROGBITS        00000000 06b76a 001362 00      0   0  1
  [16] .debug_pubnames   PROGBITS        00000000 06cacc 001267 00      0   0  1
  [17] .debug_pubtypes   PROGBITS        00000000 06dd33 0011c9 00      0   0  1
  [18] .debug_frame      PROGBITS        00000000 06eefc 000220 00      0   0  4
  [19] .comment          PROGBITS        00000000 06f11c 000013 01  MS  0   0  1
  [20] .symtab           SYMTAB          00000000 06f130 0023a0 10     22 527  4
  [21] .shstrtab         STRTAB          00000000 0714d0 0000ee 00      0   0  1
  [22] .strtab           STRTAB          00000000 0715be 00065a 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  p (processor specific)

Nothing too large here (largest is .text.dummy with 256kB). However, what is bizzare is the following:

$ riscv64-unknown-elf-objcopy -O binary -R .stack r-riscv-blink r-riscv-blink.bin
$ ls -lh r-riscv-blink{,.bin}
-rwxr-xr-x 2 tom tom 456K Jul 25 16:08 r-riscv-blink*
-rwxr-xr-x 1 tom tom 259K Jul 25 16:22 r-riscv-blink.bin*

This looks much better. Is the stack section somewhat broken? Version info:

$ riscv64-unknown-elf-objcopy --version
GNU objcopy (GNU Binutils) 2.34
Copyright (C) 2020 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.

[crepererum]

Technical Details 2

Playing around with llvm-objcopy instead, the situation becomes much clearer:

$ llvm-objcopy --version
llvm-objcopy, compatible with GNU objcopy
LLVM (http://llvm.org/):
  LLVM version 10.0.1
  Optimized build.
  Default target: x86_64-pc-linux-gnu
  Host CPU: broadwell
$ llvm-objcopy -O binary -R .stack -R .bss r-riscv-blink r-riscv-blink.bin
$ ls -lh r-riscv-blink{,.bin}
-rwxr-xr-x 2 tom tom 456K Jul 25 16:08 r-riscv-blink
-rwxr-xr-x 1 tom tom 259K Jul 26 20:34 r-riscv-blink.bin

Here, .stack and .bss must be remove to not blow up the file. Looking at the readelf output again, it also becomes clear why. .bss and .stack are the only non-empty "alloc" section with an address around 10000000, while the other "alloc" sections are located around 20000000. That's exactly the 256MB delta.

Why is this not happening on C? Simply because the sections are not spread out that much:

$ readelf -S riscv-blink.elf
There are 17 section headers, starting at offset 0x1e28:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .text             PROGBITS        20040000 001000 0001b0 00  AX  0   0  4
  [ 2] .rodata           PROGBITS        200401b0 0011b0 000000 00  WA  0   0  1
  [ 3] .data             PROGBITS        10000000 0011b0 000000 00  WA  0   0  1
  [ 4] .bss              NOBITS          10000000 000000 000000 00  WA  0   0  1
  [ 5] .debug_info       PROGBITS        00000000 0011b0 00015b 00      0   0  1
  [ 6] .debug_abbrev     PROGBITS        00000000 00130b 0000f1 00      0   0  1
  [ 7] .debug_aranges    PROGBITS        00000000 001400 000060 00      0   0  8
  [ 8] .debug_ranges     PROGBITS        00000000 001460 000048 00      0   0  8
  [ 9] .debug_line       PROGBITS        00000000 0014a8 000353 00      0   0  1
  [10] .debug_str        PROGBITS        00000000 0017fb 000186 01  MS  0   0  1
  [11] .comment          PROGBITS        00000000 001981 000022 01  MS  0   0  1
  [12] .riscv.attributes RISCV_ATTRIBUTE 00000000 0019a3 00001c 00      0   0  1
  [13] .debug_frame      PROGBITS        00000000 0019c0 00006c 00      0   0  4
  [14] .symtab           SYMTAB          00000000 001a2c 000270 10     15  23  4
  [15] .strtab           STRTAB          00000000 001c9c 0000df 00      0   0  1
  [16] .shstrtab         STRTAB          00000000 001d7b 0000aa 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  p (processor specific)

So how can the section addressed be controlled?

[crepererum]

Ah, now I get it:

icebreaker-litex-examples/rust/icebesoc-pac/memory.x

Lines 1 to 13 in b5be086

	MEMORY {
	csr : ORIGIN = 0xe0000000, LENGTH = 0x00010000
	sram : ORIGIN = 0x10000000, LENGTH = 0x00020000
	spiflash : ORIGIN = 0x20000000, LENGTH = 0x01000000
	rom : ORIGIN = 0x20040000, LENGTH = 0x00fc0000
	}
	REGION_ALIAS("REGION_TEXT", spiflash);
	REGION_ALIAS("REGION_RODATA", spiflash);
	REGION_ALIAS("REGION_DATA", sram);
	REGION_ALIAS("REGION_BSS", sram);
	REGION_ALIAS("REGION_HEAP", sram);
	REGION_ALIAS("REGION_STACK", sram);

The issue is that one part is the spiflash, the other one the RAM but everything must be one chunk for the flashing process (or at least the way it is implemented). So two question now remain:

1. why does C not use the .bss and .data segment at all?
2. Can the linker script for Rust be optimized to not leave such a large gap?

[Disasm]

I observed the issue with riscv64-linux-gnu- toolchain from ArchLinux repo, but at the same time everything was ok with riscv64-unknown-elf- toolchain from SiFive.

On my machine right now:

$ cargo update  # to fix .eh_frame_hdr problem
$ cargo run --release
$ cd target/riscv32i-unknown-none-elf/release/
$ ls -lh r-riscv-blink{,.bin}
-rwxr-xr-x 2 disasm users  72K Jul 27 00:53 r-riscv-blink
-rwxr-xr-x 1 disasm users 2.4K Jul 27 00:54 r-riscv-blink.bin

Probably something is wrong with .stack section flags, but I have no idea what is the real reason and why different toolchains behave differently.

[Disasm]

This seems to be a known (and fixed) issue: rust-lang/rust#73201
Introduced in nightly-2020-05-22 by LLVM update rust-lang/rust#67759, fixed in nightly-2020-06-08 by rust-lang/rust#73072
Present in stable 1.45.0 and 1.45.1, fixed in beta 1.46.0.

[crepererum]

Thanks a lot for investigating. I'll try out the Rust beta (which by now should contain the fix) next week and will report my findings.

I can confirm that I'm using the Arch Linux toolchain, not the one SiFive provides. So if the Rust beta does not fix the issue, I will try to change that and see if that helps.

[Disasm]

Turns out that the behavior I saw before with ArchLinux toolchain is a different story... It worked like that long before May 2020. As for the issue described here, I can reproduce it with any toolchain I have.