CI test fails

[fendor]

Seemingly in the latest commits, we have flaky CI results, e.g. some tests fail occasionally. This is really annoying, as it is hard to tell whether this is your PR's fault, or someone else's.

[lukel97]

I broke these trying to get #243 in, so as penance I'll try to fix these! All the tests were passing locally for me on macOS, so I'll need to spend some time debugging CircleCI

[lukel97]

I'm not sure if #243 did break these in the end. Looks like these were already kind of flakey here and here, but something is causing them to properly fail now. Perhaps something changed on the CircleCI VM? Cache doesn't seem to have anything to do with it