Add a quickcheck crate

[huonw]

This pulls https://github.com/BurntSushi/quickcheck into the tree (:heart: @BurntSushi).

It also extends it to include a #[quickcheck] attribute like #[test] that will run quickcheck on the annotated item (both functions and statics are supported) (thanks @MrAlert):

#![feature(phase)]

extern crate quickcheck;
#[phase(syntax)] extern crate quickcheck_macros;

#[quickcheck]
fn check_sort(mut v: Vec<int>) -> bool {
    v.sort();

    v.as_slice().windows(2).all(|pair| pair[0] <= pair[1])
}

#[quickcheck]
fn check_bad_sort(mut v: Vec<int>) -> bool {
    v.sort();
    if v.len() > 5 {
        let last = v.len() - 1;
        v.as_mut_slice().swap(0, last);
    }

    v.as_slice().windows(2).all(|pair| pair[0] <= pair[1])
}

It expands to a #[test] test and so will appear in a normal --test build:

running 2 tests
test check_sort ... ok
test check_bad_sort ... FAILED

failures:

---- check_bad_sort stdout ----
    task 'check_bad_sort' failed at '[quickcheck] TEST FAILED. Arguments: ([0, 0, 0, 0, 0, 1])', /home/huon/rust/src/libquickcheck/lib.rs:482


failures:
    check_bad_sort

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured

Other changes include:

• using StrBuf internally
• documentation adjustments (especially making the code examples work with rustdoc)
• using a trait rather than an enum for functions (thanks @BurntSushi)
• lazy shrinking for integers (thanks @bjz).

[lilyball]

👍

[erickt]

@huonw: Please add Andrew Gallant to the AUTHORS.txt file.

[erickt]

Should this just be replaced with box EmptyShrinker?

[erickt]

@BurntSushi said in irc: i think i added empty_shrinker because i was having a hard time getting type inference to kick in. e.g., if i use box EmptyShrinker as Box<Shrinker<A>>, then the compiler yells at me about an unconstrained type.

[erickt]

This looks good to me, but I figure we should wait to hear from the rest of the team to see if we're okay adding another crate to rust proper.

[huonw]

Addressed review.

[gereeter]

If this is moving into rust proper, should Shrinker be removed in favor of making Box<Iterator<A>> implement Iterator<A>?

[huonw]

It probably could, but I was thinking of waiting until DST so that Iterator can just have impl Iterator<A> for Iterator<A> (or something).

In any case, it's an easy change now/later.

[gereeter]

What is the reasoning behind making this a separate crate instead of putting it in test?

[huonw]

I personally like having smaller self-contained crates, and there's no requirement for this to be in test so it might as well be outside it.

I'm happy to be convinced otherwise.

[lilyball]

I agree, there's no need to put it in the test crate.

[huonw]

Added @MrAlert's #[quickcheck] attribute (see docs or the PR description for example of use).

[huonw]

Rendered docs: http://huonw.github.io/rust/build/doc/quickcheck/index.html

[lifthrasiir]

Can we print a different string for passing checks than "ok"? Unlike normal tests, passing checks do not guarantee that it remains correct for the extended number of trials (though one can argue that the same applies to the nondeterministic tests). Something like "{:d} checks passed" seems better.

[BurntSushi]

@lifthrasiir I think the "ok" is from the regular Rust unit testing (i.e., #[test]). QuickCheck does include output that says, "Passed N tests."

[lifthrasiir]

@BurntSushi The original output posted by @huonw includes "test check_sort ... ok", which I think misleading. It seems that the tester indeed prints a separate output ("[quickcheck] Passed {:u} tests."), but it is only enabled for DEBUG logging level.

[sfackler]

Stack overflow should be generating a normal task failure. Is this bullet still true?

[huonw]

It's a hard abort, not a failure.

[lilyball]

@lifthrasiir The "ok" is definitely printed by the #[test] test-runner. I don't think there's any way around that without making nontrivial changes to libtest (on the order of defining a new function type that has a return type that allows for overriding the "ok" string). This should not block the inclusion of quickcheck.

[lifthrasiir]

@kballard Agreed. Maybe it's a separate issue that possibly benefits other use cases.

[huonw]

I'm thinking we could take a leaf out of the polymorphic-return-book of this implementation, and basically allow

enum TestResult { Pass(StrBuf), Fail(StrBuf) }

#[test] fn normal() { ... }
#[test] fn advanced() -> TestResult { ... }

And then returning a message would allow output like

running 2 tests
test check_sort ... ok ([quickcheck] passed 100 tests)
test check_bad_sort ... FAILED

failures:

---- check_bad_sort message ----
    [quickcheck] TEST FAILED. Arguments: ([0, 0, 0, 0, 0, 1])


failures:
    check_bad_sort

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured

(Just "randomly" editing the current output.)

Or, test could be extended to be quite general, with output something like

running 2 tests
test normal_pound_test ... ok
quickcheck check_sort ... passed 100 tests
quickcheck check_bad_sort ... FAILED Arguments: [0, 0, 0, 0, 0, 1]


failures:
    check_bad_sort

test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured

[lilyball]

@huonw I don't think we need truly polymorphic return type. Whichever bit of code handles #[test] can just have explicit support for fn () and fn () -> Result<StrBuf,()>. It's an attribute that's handled by the compiler, so we don't have to worry about the type system. The test crate already has 6 different kinds of test functions, so this is just adding one more.

[huonw]

Per the weekly meeting it was decided that it's probably a bit too early to really say if Rust wants this.

I'm going to close and then (eventually) submit the patches back to @BurntSushi. Anyone who has code here may wish to submit their patch themselves (to retain authorship etc.): @MrAlert, @bjz.

[lilyball]

@huonw You can submit the patches back to @BurntSushi in MBOX format, which retains all the authorship information (including timestamp and commit message). git format-patch is the tool that generates this. And in fact GitHub can generate it for you; append ".patch" to any PR URL and you'll get a single-file MBOX with all the commits, e.g. PR #14100.patch.

[huonw]

Oh, even better; I'll do that at some point in the near future.

[lilyball]

Same trick also works on the individual commits, if you want to submit only some of them (e.g. because the first commit presumably mirrors the code @BurntSushi already has).