doctests

[epage]

We talked about doctests being an area for improvement

[weihanglo]

rustdoc is responsible for both compiling doc examples and executing them as tests. Cargo current has no way to control the behavior. There are a handful of issues around this:

• Doctests are not cached and always recompile
  • Running doc tests are slow due to being always being recompiled cargo#2944
• Doctests generate a binary per test, causing a lot of extra linking
  • Rustdoc should run all doctests in one binary rust#75341
  • cargo test renders device unresponsive due to MSVC linker RAM usage cargo#12916
• Doctests are treated separately and often forgot to test
  • cargo test --all-targets does not run doc tests cargo#6669
• Compiling and running doc examples require different set of shared libs
  • cargo test --doc not load same shared libraries as cargo test --lib cargo#8531
• Configurations in Cargo are not respected
  • target.runner — RUSTC, RUSTC_WRAPPER are not respected for doctests cargo#12532
  • codegen options — Doctests do not set profile codegen options cargo#6570
• Doctests don't work on bin's, non-pub items
  • Doctests don't work in bin targets, non-public items rust#50784

Bonus

• Warnings/clippy on doctests
  • Oh rust doctest lints, where art þou? (Add a way to run clippy on doctests) rust#56232

I propose to the team to looking into the possibility for rustdoc to delegate test executions to Cargo.

[epage]

If we can put as many doctests into a single binary as possible, that would also improve compilation times

We also want to make it so its as natural to get coverage for doctests as other tests.

I propose to the team to looking into the possibility for rustdoc to delegate test executions to Cargo.

That is the direction I would want to go as well.

[weihanglo]

Yeah we can push it further.

rustdoc, as a doc comments analyzer, provides source information via a flag. Cargo invokes rustdoc with the flag, so it knows where the source is and how to compile and test it.

[GuillaumeGomez]

I'm very fine with this approach as it would remove code from rustdoc. Problem is that the code examples originally were written to allow crate writers to add code examples for their users, meaning they needed to import the crate as if it was a dependency. I think this approach is still good, but should not be the only one (for example we can't run doctest in binary crates, which is bad). If cargo can solve both issues, that'd be awesome. :)

[epage]

Crazy idea: could we change the compilation of #[doc] to generate #[test] fns for each code fence.

• If the test lib builds against the regular lib, then the import names still work
• Otherwise, the doctests would be usable on private items, having access to everything where they are located.

Challenges

• The compiler knowing enough about doctests to do this
• The existence of compile tests (maybe an unstable feature to assert on compilation state?)

[GuillaumeGomez]

Unfortunately, it would require some new attributes for it to work as you can add a lot of different checks on doctests such as a specific edition or even to ensure that the compilation fails. More information here.

[BurntSushi]

Would it be possible to resolve how the doctest should be built with a new attribute? Maybe something like internal to opt into "compile this test as a normal unit test that uses crate internal APIs."

[epage]

Unfortunately, it would require some new attributes for it to work as you can add a lot of different checks on doctests such as a specific edition or even to ensure that the compilation fails

Yeah, I was aware of and called out compile_fail. Being able to set the edition is an odd one. Where can I read up more on the use case for that? As I don't personally have a use case for it, my automatic reaction is "can this be deprecated?" but I recognize there are likely cases I'm not aware of.

Likely switching to this style of doctests would require an opt-in with it being the default in a new edition, so we could get away with changing things.

[epage]

btw rust-lang/rust#123974 is a nice improvement to performance.

[GuillaumeGomez]

It's not finished yet. A few things need to be added:

• only enable it starting 2024 edition
• remove chunking to run all tests at once (and improve even further the performance improvement) and potentially fix libtest being blocked

Once done, I plan to work on binary doctests: I want to replace them with what we do for #[test] so people will be able to finally test their own code.