Closed
Description
The purpose is to check that all of the asv jobs are running w/o error. So modifications can be caught at PR time. I think this will be possible using asv dev
rather than trying to compare commits. This could be added onto an existing job I think (so we don't have to build again).
It also might be possible to add a travis job to actually build and compare both environments, this might be possible, again I dont' know if this would take too much time and/or be too flaky. This is not meant to replace our offline benchmarks rather to catch regressions sooner.