Run tests for cross-compiled platforms in QEMU

[alexcrichton]

QEMU has some pretty nice acceleration for x86_64/x86 targets, and it can run basically everything else under the sun in an emulation mode. We should try to run all our cross-compiled targets' tests in QEMU, or at least some subset of tests (e.g. run-pass + stdtest or something like that).

Some targets to run tests for would include:

• x86_64-unknown-freebsd
• x86_64-unknown-netbsd
• arm-unknown-linux-gnueabi
• aarch64-unknown-linux-gnu

etc

We could at least start out with testing in QEMU and then eventually upgrade one day to running a bootstrap or running the compiler in QEMU if it works out well enough.

[alexcrichton]

@japaric I've been thinking a little more about this and how it relates to a lot of the cross-compiled testing you've been working on recently. I'm thinking that the ideal here would be to take a very Android-like approach:

• At the start of a build, cross compile a small "test server"
• Place this "test server" into a disk image, and boot up an entire QEMU instance (full kernel and everything).
• Once started, this QEMU VM would then run the test server, and start listening with some protocol (maybe on TCP, maybe some serial port, w/e works)
• compiletest and rustbuild ship tests to this test server with the "test protocol" we've defined
• results are shipped back and interpreted

This'd allow us to sidestep all concerns about qemu userspace emulation bugs (e.g. with threads) and give us a full-featured testing environment (whole kernel and everything). Additionally we can perhaps get a tighter control over the environment by customizing how it's installed.

Would you be interested in prototyping this? I think it could make a great addition to the test suite we run on Travis!

[japaric]

@alexcrichton Sounds good to me. I already have an idea of how to make this test server transparently work with cross test.

Would you be interested in prototyping this?

The main reason I have never attempted QEMU system emulation is that I don't know where to get e.g. a compiled ARM kernel / rootfs. I suppose that Ubuntu / Debian must have binary releases of such things but I don't know where they are hiding them.

[alexcrichton]

Yeah it's true that assembling the qemu image would be a pain. I know for the libc-test images we use for FreeBSD and NetBSD were a pain to create.

I'd hope though that there's a "base image" with some minimal install from somewhere though that we could use!

[japaric]

OK. I found where Ubuntu was hiding their stuff:

• rootfs (See "Current releases")
• kernels

There are binary releases for armhf, ppc64el and s390x. Apart from i686 and amd64. There's a rootfs for powerpc but no kernel AFAICT.

And I got a script to turn those tarballs into a QEMU image (the armhf image is around 300MB in size). Now we are just missing the test server.

[alexcrichton]

Holy cow, awesome! That was fast :)

My guess is that we could make a super simple test protocol server for now, something like:

• Spin up a TCP server
• Each test initiates a new TCP connection
• First, the binary is sent over the connection
• Next, arguments and environments are sent over the connection
• Next, the connection becomes "duplex" where data read is put on stdin and data read on stdout goes back onto the socket
• Finally the exit status is transmitted with some final frame

[japaric]

Yeah, I got a simpler test server (no stdin or arguments) working on the host but it doesn't when the server part is running inside QEMU. I could be doing the QEMU TCP redirection wrong and some sort of networking has to be done on the guest.

Here's the code (sorry, it's an unwrap-fest), if you want to give it a try.

Usage looks like this:

# host term 1
$ testd
Listening on :12345
# blocks

# host term 2
$ testc hello
Hello, world!

# host term 1
$ testd
Listening on :12345
File { name: "hello", size: 3517688 }
ExitStatus { code: Some(0), success: true }
# blocks again

I'm launching QEMU like this:

$ qemu-system-arm \
    -M virt \
    -append "root=/dev/vda rw" \
    -hda armhf.qcow2 \
    -kernel vmlinuz-armhf \
    -m 1024 \
    -monitor telnet:127.0.0.1:1234,server,nowait \
    -no-reboot \
    -nographic \
    -net nic,model=virtio \
    -net user,hostfwd=tcp::12345-:12345 \
    -serial stdio

And it gets stuck like this

# Guest (inside QEMU)
$ testd
Listening on :12345
# blocks

$ Host
$ testc hello
# blocks

AFAICT, the client does open a connection and sends the data because QEMU is listening on port 12345 but QEMU is not passing that data to the guest for some reason.

[alexcrichton]

Hm some of the QEMU documentation alludes to:

Redirect incoming TCP or UDP connections to the host port
hostport to the guest IP address guestaddr on guest port
guestport. If guestaddr is not specified, its value is
x.x.x.15 (default first address given by the built-in DHCP
server). By specifying hostaddr, the rule can be bound to a
specific host interface. If no connection type is set, TCP is
used. This option can be given multiple times.

Maybe guesaddr should be explicitly specified in hostfwd to 127.0.0.1 because that's what the server is listening on? Alternatively you could switch the listening address to 0.0.0.0 on the testd side of things.

[japaric]

Maybe guesaddr should be explicitly specified in hostfwd to 127.0.0.1

I had already tried explicitly listing 127.0.0.1 in several places. It didn't help

Alternatively you could switch the listening address to 0.0.0.0 on the testd side of things.

I just tested this ... no dice.

QEMU does report the port forwarding is enabled though:

$ telnet 127.0.0.1 1234
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
QEMU 2.8.0 monitor - type 'help' for more information
(qemu) info usernet
VLAN 0 (user.0):
  Protocol[State]    FD  Source Address  Port   Dest. Address  Port RecvQ SendQ
  TCP[SYN_SENT]      13       127.0.0.1 12345       127.0.0.1 12345     0     0
  TCP[HOST_FORWARD]  11               * 12345       127.0.0.1 12345     0     0

Does ... TCP not work in Linux if you forget to initialize some service? I tried systemctl start systemd-networkd.

I also discarded this being an architecture specific problem because I also tried running testd in a x86_64 guest whose image was generated with the script linked above.

[luser]

For the record, the architecture you've outlined would work well for testing on devices and such as well, which would be nice. We have something similar we used for Firefox testing on devices in the past called the SUTAgent, although it's been deprecated in favor of using ADB directly for Android testing on devices or in emulators now.

[alexcrichton]

@japaric hm I poked around a bit and could reproduce everything you mentioned as well. I was seeing some odd errors though about a readonly filesystem and I added rw to the append option passed to qemu and it made those go away at least. I wasn't able to get much further than that, though.

I wonder if there's some kernel option that's not being passed or something like that? I have very little experience with qemu network and barely understand it whenever it works, so it could be an obvious problem that we're just missing...

[alexcrichton]

Ok I've spent what is now an embarassing amount of time trying to get this work. Along the way I think I'm learning things about Linux, but who knows!

@japaric so I think what's going on in your gist is that the guest doesn't have networking configured. Now that being said I have no idea how to configure the network with such a minimal image. As a result I opted to try a different strategy, outlined in a gist. Everything's a bit of a mess, so I figured I'd list a few things in detail:

• I opted to take a different boot strategy, e.g. boot-to-busybox. Basically when playing with networking a busybox-booted image was the first thing I got networking working with.
• I could never figure out -M virt, so I opted to compile the kernel from source. This also mean that I'm now using vexpress-a15 as an emulation machine. I previously tried versatilepb but it doesn't support more than 256M of ram which probably won't suit our needs. TBH I don't know what -M virt is, and vexpress-a15 was also otherwise the first thing that worked.
• I don't think we'll need ext2 images and stuff. Turns out we can just shove everything into an initrd image which can be created with the standard cpio tool.

So given all that the gisted Dockerfile should be enough to boot up a guest which spawns testd on startup. The port forwarding should be set up so if you're in the docker container you can then connect to that port. I've been testing this with:

# in one shell
docker run -it $container

# in another shell, when booted
docker exec -it `docker ps -l -q` testd/target/release/testc testd/target/arm-unknown-linux-gnueabihf/release/hello

Unfortunately it's not working yet. I believe the reason is that reads of /dev/random are blocking. The server (current testd) uses TempDir which transitively uses rand in Rust which will eventually bottom out to /dev/random or the equivalent syscall. Unfortunately nothing I've been able to do has gotten the entropy pool to fill up. I don't think we can gloss over this issue for the server as well because tests will need access to randomness and at this point I don't think it'll ever fill up.

Figured I'd at least give a bit of an update. For the number of hours I've put into this so far it seems I've only made a tiny bit of progress!

[alexcrichton]

Oh also apologies for the state of the gist, everything's a mess :(. GitHub apparently loses directory structure when gisting, but it should have a relatively self-explanatory layout.