sf visual tests are unstable

[lionel-]

Probably because of the sf dependencies:

✖ |  0 1     | coord_sf [1.0 s]
────────────────────────────────────────────────────────────────────────────
test-coord_sf.R:10: failure: multiplication works
Figures don't match: sf-polygons.svg

────────────────────────────────────────────────────────────────────────────


✖ |  1 1     | geom-sf [1.1 s]
────────────────────────────────────────────────────────────────────────────
test-geom-sf.R:13: failure: geom_sf draws correctly
Figures don't match: north-carolina-county-boundaries.svg

────────────────────────────────────────────────────────────────────────────

[clauswilke]

@lionel- I think it's almost impossible to guarantee that no visual test ever changes as some other package is changed. We just saw (#2824) that a small update in scales broke many visual tests, and similar scenarios can continue to arise.

Could we have a conversation of what the key issues are that arise from this problem and how they could be mitigated? I see three issues, ordered from less to more severe:

1. If different developers use different package versions, the visual tests may keep breaking for them, or different developers may end up repeatedly committing different versions of the reference image for a test.
2. It may not be possible to have one single reference image for all R versions on which ggplot2 is tested.
3. Visual tests may eventually break if run on CRAN as other packages are updated.

For point 1, I think the developers could just agree on a reasonable set of packages to use (e.g., whatever is most recent on CRAN). For point 2, I'm not sure. Maybe run the visual tests only for one R version? Or alternatively, find a way to have different reference images for different R versions? But that could get very tedious. For point 3, I think the right answer is to never run visual tests of CRAN. The risk of something minor breaking for innocuous reasons is too high, and we don't want CRAN threatening to pull ggplot2 just because some visual test failed.

Are there any other concerns I've missed?

[lionel-]

I think we should just fail silently when dependencies are outdated (and we should add scales to deps of ggplot2 plots). This way we don't overwhelm CRAN with spurious failures, but we still display helpful messages to the developer. Packages should always write tests for internal routines and data structures and those tests would fail verbosely on CRAN.

I think the conclusion of all this is that vdiffr is more of a monitoring mechanism than a unit testing tool.

[lionel-]

Instead of a complicated dependency mechanism maybe we should fail silently by default unless an environment variable is set? Then we can set it in Travis to get verbose failures in pull requests.

[lock]

This old issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with reprex) and link to this issue. https://reprex.tidyverse.org/