CI: how to ensure certain tests have been run

[jorisvandenbossche]

From a discussion in #26103: for some time the clipboard tests have not been running, without that we actually noticed it (apart from that something in the doc build failed for a similar reason).

You can check the skipped tests (but they are only in there if the tests were ran in the first place, which depends on the selection marker).

@jreback suggested printing something (for the specific case of the clipboard tests), but although we have -s that didn't seem to work

In general, it might be nice to have some overview of the tests that have run. Using pytest -v however shows every test on a single line, and is probably too much. Something in between like showing each file (and the tests as dots next to each file) would be nice instead of only dots. Anybody know if that is possible?

[TomAugspurger]

I’d be interested in a GitHub bot or an azure step that analyzes the uploaded test coverage files. We could have a list of tests that must be run, or the CI fails. And we can maybe note if a bunch of previously passing tests are now skipped.