[llvm-cov] 'llvm-cov show' output for template functions with lots of instantiations is overwhelming

[bartlettroscoe]

Description

As of LLVM 20.1, the llvm-cov show command produces an overwhelming amount of output in the HTML file if there are a lot of template instantiations. Even if there are not a large number of template instantiations, the HTML file is still very hard to follow. For example, for the Trilinos C++ file Teuchos_RCP.hpp, if you only build and run the Teuchos test suite, you get a fairly reasonable output HTML file Teuchos_RCP.html (see reduced version for just the lines 121-149 attached) which looks like:

---

---

See full image here. As you see in that image, just with three instantiations, the main template function is obscured by the template instantiations.

But when you have lot of template instantiations, the HTML file becomes completely unreadable. When you build and run the entire Trilinos test suite, the generated file Teuchos_RCP.html becomes about impossible to follow with over 150 instantiations (see reduced version for just the lines 121-149 attached) which looks like:

---

---

(I can't show all of the branches for even the first if() statement!.)

And if you consider that fact that there are over 150 instantiations of this function, trying to figure out where you are in the source file is about impossible.

Now, compare that to the gcovr produced coverage for LLVM clang using the gcov-comparible coverage using --coverage and the tool llvm-cov gcov for the same builds of and test suites. Running gcovr -r <src-dir> --object-directory . --gcov-executable="llvm-cov gcov" --txt-metric branch --gcov-ignore-parse-errors=suspicious_hits.warn_once_per_file --html-nested <output-file> produces coverage for Teuchos_RCP.hpp (see attached) that looks like:

---

---

Notice how you don't get any clutter for the template instantiations at all. The file is very easy to follow. However, the branch coverage is confusing and may not even be correct, a shown by:

---

---

However, those branches don't make a lot of sense to me. The description of branch coverage with llvm-cov show with the source-based coverage is much more logical and direct (IMHO). (Which is one reason I want to use the newer source-based coverage over the older gcov-compatible coverage and gcovr.)

Suggested solution

We can have our cake and eat it too. All we need to do is to have llv-cov show wrap the branch and line coverage details inside of <details> ... </details> blocks. For example, taking the direct output from llvm-cov show shown here and adding <details> ... </details> blocks by applying this diff gives you the new HTML file shown here showing:

---

---

NOTE: I manually computed the combined branch coverage counts for the different instantiations. The real implementation would do this automatically.

Now you can expand and contract the branch coverage like:

---

---

and:

---

---

And you can expand the template instantiations like:

---

---

Now, applying this to the Teuchos_RCP.hpp.html file that has over 130 template instantiations you get the file the attached file which looks like:

---

---

And as you can see, the page Teuchos_RCP.hpp is very easy to read. And you can expand whatever details you want to see for specific instantiations.

All files

A tarred-gziped directory of all of the files mentioned above, and more, is attached.

[bartlettroscoe]

NOTE: I could likely write a post-processing tool that puts in these <details> ... </details> blocks, but it would be much better if llvm-cov show just put them in by default automatically (or at least gave us an option to insert them).

Because the current way that llvm-cov show generates these HTML files for multiple template instantiations, I think we would have to go with gcovr and the gcov-compatible coverage support. Otherwise, I don't think most developers would be able to navigate these generated HTML files for larger template functions and/or numerous template instantiations.

[bartlettroscoe]

Also, while we are at it, the links in the branch coverage list in the main template function should link to the branch line in the template instantiation block, not to the line in the template function. For example, the hyper link for Branch (131:7) [True: 3.63k, False: 0] in the parent template function:

---

---

should link to:

---

---

NOT to:

---

---

as it currently does.

[chapuni]

I suppose per-instantiation branch coverages can be seen in "template expansion subviews". But it would emit too large HTML to crash browsers.

You suggest adding per-instantiation branches. Wouldn't you care about line counts? I thinks this issue as how to show per-instantiation information.

[bartlettroscoe]

I suppose per-instantiation branch coverages can be seen in "template expansion subviews". But it would emit too large HTML to crash browsers.

It already does, and it does not crash the browsers (but it does take a lot of memory).

You suggest adding per-instantiation branches. Wouldn't you care about line counts? I thinks this issue as how to show per-instantiation information.

@chapuni, I think there may be some confusion here. The command llvm-cov show already creates all of this per-instantiation information. The file llvm-cov_show.Teuchos_RCP.hpp.all_packages.html in the tarred/gziped directory llvm-cov_show_files.tar.gz was generated with llvm-cov show with LLVM 20.1. It is 92M in size! The original source file Teuchos_RCP.hpp is only 20k. You can see that it contains branch coverage for all of the template instantiations (go to the line llvm-cov_show.Teuchos_RCP.hpp.all_packages.html#L131 and you will see that). This file is unreadable (IMHO).

All I am suggesting with this Issue is to put in <details> ... </details> blocks to hide his massive glut of template instantiation information in the default view. If we put in those <details> ... </details>, the file would be much more readable (and it might even load faster?).

Does that make sense?

[bartlettroscoe]

@EugeneZelenko, technically, this is also a suggestion for an enhancement (see "Suggested solution" above to add <details> ... </details> blocks).

[bartlettroscoe]

NOTE: There are template functions in Trilinos that have upwards of 1k instantiations! The current implementation is not scalable.