debuginfo: Add support for split-debuginfo on platforms that allow it

[michaelwoerister]

DWARF v5 will standardize something sometimes called "Split DWARF" or "debuginfo fission". The gist of it is this: Debuginfo can be very large (gigabytes) and can contain lots of relocations, so the linker will spend a lot of time copying and relocating it into the final executable. "Split DWARF" is an approach that allows to basically skip this linker step: Since debuginfo is already located in the individual object files generated during compilation, why not just "link" to the debuginfo in there and let the debugger do any relocations on demand. This can potentially mean a drastic reduction of compile times for builds with debuginfo.

LLVM already supports this on Linux, as far as I know, and it might be a good fit for incremental compilation where the linker could easily become the most time-consuming step (although that might be nixed by using gold's incremental linking feature).

It might also be interesting for providing pre-built debuginfo separately via *.dwp files.

[nagisa]

Technically it is already possible to have “split” DWARF debug info, even without v5, which you can load into gdb.

The only problem is generating it separately in the first place, because currently the only way to get split debug info to the best of my knowledge, is to use objcopy --only-keep-debug and then strip the original library/executable.

[michaelwoerister]

The only problem is generating it separately in the first place, because currently the only way to get split debug info to the best of my knowledge, is to use objcopy --only-keep-debug and then strip the original library/executable.

How does gdb know where to find the debuginfo in that case? With the DWARF 5 approach, the 'skeleton' debuginfo entries store the path to the object files.

[nagisa]

You use command like add-symbol-file. Sure not the most seamless experience, but, like I said, it at least works.

[tromey]

How does gdb know where to find the debuginfo in that case?

There are two ways, but the better of the two is the build-id feature. See the docs. This is what all the distros use.

However, this form of splitting is very different from Fission. Fission trades off link time speed for some debugger performance (I believe, I haven't used it in anger). Build-id is more about being able to split off the debuginfo for separate packaging -- and, contra Fission, it slows down the development cycle, as the splitting is a separate step.

It's not always appropriate to use Fission so this would have to be a compile-time flag of some kind.

[michaelwoerister]

@tromey Thanks for the background info! Would you agree that Fission with .dwp packages would be the cleaner way of distributing debuginfo separately? I know, this feature just wasn't available before, so distros had to do it differently. But now that it is there (in some places at least :)), it seems like the superior approach.

It's not always appropriate to use Fission so this would have to be a compile-time flag of some kind.

Yes, that's how I envisioned it.

[tromey]

Would you agree that Fission with .dwp packages would be the cleaner way of distributing debuginfo separately?

They are maybe mildly cleaner (but maybe not); but the approach currently taken by the distros has the advantage that all the tools already work with it; whereas nobody did the work to make everything use Fission. Also the distros, IME, were more space-sensitive and less link-time-sensitive than developers; leading to dwz instead, which is somewhat tied into the build-id and packaging schemes.

[michaelwoerister]

@tromey That is good to know. Thanks!

[alexcrichton]

Adding a cc from #47240 to this as well. OSX supports this by default but we're actually going out of our way to undo the compile time win by running dsymutil by default. It looks like we don't need to though and instead need to ensure that we leave the object files without deleting them. That's probably also relevant for this where we'll need to avoid deleting some intermediate artifacts!

[rocallahan]

LLVM already supports this on Linux, as far as I know,

To be clear: there are two flavours of split-DWARF: the pre-standard GNU flavour and the DWARF5 standardized flavour, and they are different. (Par for the course due to DWARF's broken development model :-(.) LLVM/gcc/gdb support the former but not the latter and probably there is not yet any tool support for the DWARF5 standard flavour. See http://lists.dwarfstd.org/pipermail/dwarf-discuss-dwarfstd.org/2017-February/004325.html. So your choice is between using the GNU flavour or waiting some unknown length of time for the standard flavour to be usable.

Using split-DWARF would create inconvenience for some users. E.g. we move Rust binaries around between machines/containers and sometimes need to debug binaries on machines other than where they were built. Split-DWARF would mean we have to move around the object files as well for debugging to work, which would be less efficient than just not splitting. So we'd want split-DWARF to be an option we could turn off (and linking performance without split-DWARF would continue to matter for us).