Testing wishlist

[jonmmease]

This issue is to discuss ideas and plans for improving testing for plotly.py. I started thinking about this more while working on the CircleCI 2.0 migration in #1109.

In no particular order:

• Migrate to pytest

  • parameterization and lightweight fixtures are really useful, and pytest is already in use for the validator tests in _plotly_utils/tests
  • pytest can run most nosetests as is, so I don't expect this to be a lot of work (https://docs.pytest.org/en/latest/nose.html)

• Reintroduce pytest-based code coverage reporting (https://pytest-cov.readthedocs.io/en/latest/)

• Test the code generation logic on Python 3.6+. Make sure generation runs and produces no change relative to the committed code.

• Add a Python 3.6+ workflow that runs code generation against Plotly.js master, and then runs all of the tests. We wouldn't hold up PRs on this, but it would help us catch compatibility issues before Plotly.js releases. Maybe it just runs periodically on master.

• Add orca integration tests to the CI test suite.

• Add some selenium-based integration tests for FigureWidget in the classic notebook and in JupyterLab.

• Add OS X and Windows CI tests. This will be especially helpful for the orca server process integration (Orca integration for static image export #1105) since the process management logic has some OS-dependent considerations.

cc @chriddyp @cldougl @nicolaskruchten @Kully

[matanox]

Maybe it's not migrated to pytest yet, at least #3670 may give that impression since there's a small interoperability issue there. Sorry to advertise that ticket like this, but would be great to have that interoperability with pytest, so that user tests in complicated data science projects may cover calling plotly express api up to the final show() step, or when using something like Xvfb for end-to-end headless testing, to make sure that visualization steps aren't broken.

[nicolaskruchten]

We do use pytest for testing Plotly Express and the rest of Plotly.

[gvwilson]

Hi - we are currently trying to tidy up Plotly's public repositories to help us focus our efforts on things that will help users most. Since this issue has been sitting for several years, so I'm going to close it. If it's still a concern, we'd be grateful if you could open a new issue (with a short reproducible example if appropriate) so that we can add it to our backlog. Thanks for your help - @gvwilson