Create a UI-like test suite for Cargo

[alexcrichton]

The src/test/ui tests in the rust-lang/rust repository I think have been wildly successful. I believe this is because:

• The tests are very easy to add (just drop a file somewhere)
• Each test's output is exhaustively checked all the time.
• Output assertions can be automatically updated
  • Makes it easy to write a test, generate the expected output, and review it to make sure it's ok
  • Also makes it very easy to spot output differences in online diffs
  • Also makes it very easy to blanket update the entire test suite
• Tests can have lightweight configuration (// directives at the top) to customize how the test is run.

Cargo hasn't historically done this, but thinking recently I'm wondering if we could actually implement this? It would solve a number of downsides about the current test suite, and I think it'd be awesome if we could migrate towards a system like this.

This is soemwhat vague and just an idea though. I don't think we'd be able to migrate 100% of the tests, but we sure do have a lot of tests that are basically "set the project up, run a build, make sure the output looks ok". We might even be able to move some of the more advanced tests into this format eventually, but there's no reason to migrate 100% of the tests I think if some suffice.

[ehuss]

I think the idea of using compiletest-UI-style testing is intriguing. It's tough because just adding/changing a rustc flag would cause a lot of churn for all -v tests. Also, the order of output is often nondeterministic. And there's often a bunch of hashes that somehow need to be carefully ignored. Something like --bless would help, but reviewing a PR that changes hundreds of tests would not be possible.

I recently added with_stderr_line_without which is a bit awkward to use and also dangerous (like does_not_contain). These tests could assert the entire output, but it changes a little too often, and matching command-line flags is difficult.

That being said, I think it would be great to experiment with.

[alexcrichton]

That's true yeah that if we have to regenerate the expectations every night with a new rustc then this woud be completely untenable. I believe the rustc ui tests do some degree of normalization such as stripping the cwd from the output, line numbers, etc. I suspect we could do similar to paste over issues with hashes, rustc flags that are uninteresting, etc. If the problem persists though then this isn't worth it!

This is definitely a large-ish undertaking for Cargo. I'm not certain we should do this and if someone's interested in picking this up it'd largely be an evaluation phase to see if it can work at all.

[fluffysquirrels]

Cargo integration tests already have a lot of the requirements ticked, I'd say. The main difference seems to be that it'd be easier to update all the output of the whole test suite. How common or likely is that?

[alexcrichton]

It doesn't happen too too often, but it's definitely a pain to do so when it happens.

I definitely agree that today's system checks a lot of boxes so we don't have a ton of motivation to change it, but lots of tests don't assert much about their output (asserting contains rather than the whole output) and could possibly be improved if we had a UI test suite like this.

[matklad]

An interesting middle ground is to allow to define the whole (multifile) input fixture as a string literal, and the output result is an "updatable" string literal

• https://github.com/rust-analyzer/rust-analyzer/blob/0f7961d5570f17d6c2098ab11d2a3bcbbfb84ff6/crates/ra_hir_ty/src/tests/traits.rs#L13-L34
• https://github.com/rust-analyzer/rust-analyzer/blob/0f7961d5570f17d6c2098ab11d2a3bcbbfb84ff6/crates/ra_hir_ty/src/tests/traits.rs#L250-L273

The drawback here is that updating the tests is more fun -- you need some macro magic with file!() and line!() to write a test which updates its own source code. You also loose ability to cd in the test directory and just run cargo to try things out.

The main benefit is that you don't have hundreds of files lying. The problem with many files is that they pollute t on GitHub, fuzzy file serarch in the editor, ctags/symbols search, etc.

Additionally, creating several small files might actually be more annoying than modifyng a single string in one place. It's also nice to have a literal #[test] fn for each test which you can run from an IDE :)

For rust-analyzer, I am pretty convinced that "inline" tests are better, but for cargo (due to the cd test && cargo bulid) I think separate files/dirs might make more sense.

[epage]

We have more of the items covered today (including auto-updating) with the use of snapbox. I'd love to extend it to auto-update in-memory strings but I want a reliable way of finding them first.