Skip to content

Test against JSON summaries (and bugfixes) #134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 22 commits into from
Feb 9, 2022
Merged

Test against JSON summaries (and bugfixes) #134

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
35404a0
Helpers to diff JSON summary
ConorMacBride Feb 3, 2022
5f66e2e
Runner for subtests
ConorMacBride Feb 3, 2022
9f7deb0
Support for version specific patches
ConorMacBride Feb 4, 2022
2854e03
Only show match errors at end of test
ConorMacBride Feb 4, 2022
7e95ae8
Hash libraries for baseline and result
ConorMacBride Feb 4, 2022
2796fad
Add subtests
ConorMacBride Feb 4, 2022
49fcd8d
Generate baseline with full path for tox
ConorMacBride Feb 4, 2022
cd45753
Skip tests without required libraries
ConorMacBride Feb 5, 2022
f82256d
Fix bug in html without hash comparison
ConorMacBride Feb 5, 2022
ea144f7
Add subtest baselines
ConorMacBride Feb 5, 2022
7e65619
Add mpl33 to mpl35 baselines
ConorMacBride Feb 5, 2022
2d035e1
Add draft baseline summaries
ConorMacBride Feb 5, 2022
ab4bd47
Document `--mpl-baseline-relative`
ConorMacBride Feb 5, 2022
b0f62ba
Clarify behaviour
ConorMacBride Feb 5, 2022
de57cab
Update summaries for new hash and hybrid logs
ConorMacBride Feb 6, 2022
a5272a6
Update tests for new status_msg
ConorMacBride Feb 6, 2022
a2fc28b
Record image filenames during test
ConorMacBride Feb 6, 2022
188a08d
Fix bugs and refactor compare_image_to_hash_library
ConorMacBride Feb 6, 2022
e44c0c1
Update wording in log message
ConorMacBride Feb 6, 2022
03d1f76
Convert status_msg to regex
ConorMacBride Feb 6, 2022
1cb3d4e
Improved support for hash comparison only HTML summary
ConorMacBride Feb 6, 2022
33338ba
Always record image paths as POSIX
ConorMacBride Feb 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 33 additions & 19 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,8 @@ directly in the right directory.
With a Hash Library
^^^^^^^^^^^^^^^^^^^

Instead of comparing to baseline images, you can instead compare against a json
library of sha256 hashes. This has the advantage of not having to check baseline
Instead of comparing to baseline images, you can instead compare against a JSON
library of SHA-256 hashes. This has the advantage of not having to check baseline
images into the repository with the tests, or download them from a remote
source.

Expand All @@ -91,8 +91,11 @@ Hybrid Mode: Hashes and Images
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

It is possible to configure both hashes and baseline images. In this scenario
the hashes will be compared first, with the baseline images only used if the hash
comparison fails.
only the hash comparison can determine the test result. If the hash comparison
fails, the test will fail, however a comparison to the baseline image will be
carried out so the actual difference can be seen. If the hash comparison passes,
the comparison to the baseline image is skipped (unless **results always** is
configured).

This is especially useful if the baseline images are external to the repository
containing the tests, and are accessed via HTTP. In this situation, if the hashes
Expand All @@ -104,7 +107,7 @@ without having to modify the external images.
Running Tests
^^^^^^^^^^^^^

Once tests are written with either baseline images or a hash library to compare
Once tests are written with baseline images, a hash library, or both to compare
against, the tests can be run with::

pytest --mpl
Expand All @@ -118,12 +121,15 @@ Generating a Test Summary
^^^^^^^^^^^^^^^^^^^^^^^^^

By specifying the ``--mpl-generate-summary=html`` CLI argument, a HTML summary
page will be generated showing the result, log entry and RMS of each test,
and the hashes if configured. The baseline, diff and result image for each
failing test will be shown. If **Results always** is configured
(see section below), images for passing tests will also be shown.
If no baseline images are configured, just the result images will
be displayed.
page will be generated showing the test result, log entry and generated result
image. When in the (default) image comparison mode, the baseline image, diff
image and RMS (if any), and tolerance of each test will also be shown.
When in the hash comparison mode, the baseline hash and result hash will
also be shown. When in hybrid mode, all of these are included.

When generating a HTML summary, the ``--mpl-results-always`` option is
automatically applied (see section below). Therefore images for passing
tests will also be shown.

+---------------+---------------+---------------+
| |html all| | |html filter| | |html result| |
Expand Down Expand Up @@ -188,28 +194,36 @@ running tests by running ``pytest`` with::

pytest --mpl --mpl-baseline-path=baseline_images

This directory will be interpreted as being relative to where the tests
are run. In addition, if both this option and the ``baseline_dir``
This directory will be interpreted as being relative to where pytest
is run. However, if the ``--mpl-baseline-relative`` option is also
included, this directory will be interpreted as being relative to
the current test directory.
In addition, if both this option and the ``baseline_dir``
option in the ``mpl_image_compare`` decorator are used, the one in the
decorator takes precedence.

Results always
^^^^^^^^^^^^^^

By default, result images are only generated for tests that fail.
By default, result images are only saved for tests that fail.
Passing ``--mpl-results-always`` to pytest will force result images
to be generated for all tests, even for tests that pass.
If a baseline image exists, a diff image will also be generated.
All of these images will be shown in the summary (if requested).
to be saved for all tests, even for tests that pass.

When in **hybrid mode**, even if a test passes hash comparison,
a comparison to the baseline image will also be carried out,
with the baseline image and diff image (if image comparison fails)
saved for all tests. This secondary comparison will not affect
the success status of the test.

This option is useful for always *comparing* the result images again
This option is useful for always *comparing* the result images against
the baseline images, while only *assessing* the tests against the
hash library.
If you only update your baseline images after merging a PR, this
option means that the generated summary will always show how the
PR affects the baseline images, with the success status of each
test (based on the hash library) also shown in the generated
summary.
summary. This option is applied automatically when generating
a HTML summary.

Base style
^^^^^^^^^^
Expand Down
135 changes: 45 additions & 90 deletions pytest_mpl/plugin.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ def pytest_addoption(parser):
group.addoption('--mpl-results-path', help=results_path_help, action='store')
parser.addini('mpl-results-path', help=results_path_help)

results_always_help = ("Always generate result images, not just for failed tests. "
results_always_help = ("Always compare to baseline images and save result images, even for passing tests. "
"This option is automatically applied when generating a HTML summary.")
group.addoption('--mpl-results-always', action='store_true',
help=results_always_help)
Expand Down Expand Up @@ -272,7 +272,7 @@ def __init__(self,
if len(unsupported_formats) > 0:
raise ValueError(f"The mpl summary type(s) '{sorted(unsupported_formats)}' "
"are not supported.")
# Ignore `results_always` and always save result images for HTML output
# When generating HTML always apply `results_always`
if generate_summary & {'html', 'basic-html'}:
results_always = True
self.generate_summary = generate_summary
Expand Down Expand Up @@ -431,7 +431,7 @@ def compare_image_to_baseline(self, item, fig, result_dir, summary=None):

test_image = (result_dir / "result.png").absolute()
fig.savefig(str(test_image), **savefig_kwargs)
summary['result_image'] = '%EXISTS%'
summary['result_image'] = test_image.relative_to(self.results_dir).as_posix()

if not os.path.exists(baseline_image_ref):
summary['status'] = 'failed'
Expand All @@ -447,7 +447,7 @@ def compare_image_to_baseline(self, item, fig, result_dir, summary=None):
# copy to our tmpdir to be sure to keep them in case of failure
baseline_image = (result_dir / "baseline.png").absolute()
shutil.copyfile(baseline_image_ref, baseline_image)
summary['baseline_image'] = '%EXISTS%'
summary['baseline_image'] = baseline_image.relative_to(self.results_dir).as_posix()

# Compare image size ourselves since the Matplotlib
# exception is a bit cryptic in this case and doesn't show
Expand All @@ -472,7 +472,8 @@ def compare_image_to_baseline(self, item, fig, result_dir, summary=None):
else:
summary['status'] = 'failed'
summary['rms'] = results['rms']
summary['diff_image'] = '%EXISTS%'
diff_image = (result_dir / 'result-failed-diff.png').absolute()
summary['diff_image'] = diff_image.relative_to(self.results_dir).as_posix()
template = ['Error: Image files did not match.',
'RMS Value: {rms}',
'Expected: \n {expected}',
Expand All @@ -488,9 +489,7 @@ def load_hash_library(self, library_path):
return json.load(fp)

def compare_image_to_hash_library(self, item, fig, result_dir, summary=None):
new_test = False
hash_comparison_pass = False
baseline_image_path = None
if summary is None:
summary = {}

Expand All @@ -505,87 +504,58 @@ def compare_image_to_hash_library(self, item, fig, result_dir, summary=None):

hash_library = self.load_hash_library(hash_library_filename)
hash_name = self.generate_test_name(item)
baseline_hash = hash_library.get(hash_name, None)
summary['baseline_hash'] = baseline_hash

test_hash = self.generate_image_hash(item, fig)
summary['result_hash'] = test_hash

if hash_name not in hash_library:
new_test = True
if baseline_hash is None: # hash-missing
summary['status'] = 'failed'
error_message = (f"Hash for test '{hash_name}' not found in {hash_library_filename}. "
f"Generated hash is {test_hash}.")
summary['status_msg'] = error_message
else:
summary['baseline_hash'] = hash_library[hash_name]
summary['status_msg'] = (f"Hash for test '{hash_name}' not found in {hash_library_filename}. "
f"Generated hash is {test_hash}.")
elif test_hash == baseline_hash: # hash-match
hash_comparison_pass = True
summary['status'] = 'passed'
summary['status_msg'] = 'Test hash matches baseline hash.'
else: # hash-diff
summary['status'] = 'failed'
summary['status_msg'] = (f"Hash {test_hash} doesn't match hash "
f"{baseline_hash} in library "
f"{hash_library_filename} for test {hash_name}.")

# Save the figure for later summary (will be removed later if not needed)
test_image = (result_dir / "result.png").absolute()
fig.savefig(str(test_image), **savefig_kwargs)
summary['result_image'] = '%EXISTS%'
summary['result_image'] = test_image.relative_to(self.results_dir).as_posix()

if not new_test:
if test_hash == hash_library[hash_name]:
hash_comparison_pass = True
summary['status'] = 'passed'
summary['status_msg'] = 'Test hash matches baseline hash.'
else:
error_message = (f"Hash {test_hash} doesn't match hash "
f"{hash_library[hash_name]} in library "
f"{hash_library_filename} for test {hash_name}.")
summary['status'] = 'failed'
summary['status_msg'] = 'Test hash does not match baseline hash.'

# If the compare has only been specified with hash and not baseline
# dir, don't attempt to find a baseline image at the default path.
if not hash_comparison_pass and not self.baseline_directory_specified(item) or new_test:
return error_message
# Hybrid mode (hash and image comparison)
if self.baseline_directory_specified(item):

# If this is not a new test try and get the baseline image.
if not new_test:
baseline_error = None
baseline_summary = {}
# Ignore Errors here as it's possible the reference image dosen't exist yet.
try:
baseline_image_path = self.obtain_baseline_image(item, result_dir)
baseline_image = baseline_image_path
if baseline_image and not baseline_image.exists():
baseline_image = None
# Get the baseline and generate a diff image, always so that
# --mpl-results-always can be respected.
# Skip image comparison if hash matches (unless `--mpl-results-always`)
if hash_comparison_pass and not self.results_always:
return

# Run image comparison
baseline_summary = {} # summary for image comparison to merge with hash comparison summary
try: # Ignore all errors as success does not influence the overall test result
baseline_comparison = self.compare_image_to_baseline(item, fig, result_dir,
summary=baseline_summary)
except Exception as e:
baseline_image = None
baseline_error = e
for k in ['baseline_image', 'diff_image', 'rms', 'tolerance', 'result_image']:
summary[k] = summary[k] or baseline_summary.get(k)

# If the hash comparison passes then return
if hash_comparison_pass:
except Exception as baseline_error: # Append to test error later
baseline_comparison = str(baseline_error)
else: # Update main summary
for k in ['baseline_image', 'diff_image', 'rms', 'tolerance', 'result_image']:
summary[k] = summary[k] or baseline_summary.get(k)

# Append the log from image comparison
r = baseline_comparison or "The comparison to the baseline image succeeded."
summary['status_msg'] += ("\n\n"
"Image comparison test\n"
"---------------------\n") + r

if hash_comparison_pass: # Return None to indicate test passed
return

if baseline_image is None:
error_message += f"\nUnable to find baseline image for {item}."
if baseline_error:
error_message += f"\n{baseline_error}"
summary['status'] = 'failed'
summary['status_msg'] = error_message
return error_message

summary['baseline_image'] = '%EXISTS%'

# Override the tolerance (if not explicitly set) to 0 as the hashes are not forgiving
tolerance = compare.kwargs.get('tolerance', None)
if not tolerance:
compare.kwargs['tolerance'] = 0

comparison_error = (baseline_comparison or
"\nHowever, the comparison to the baseline image succeeded.")

error_message = f"{error_message}\n{comparison_error}"
summary['status'] = 'failed'
summary['status_msg'] = error_message
return error_message
return summary['status_msg']

def pytest_runtest_setup(self, item): # noqa

Expand Down Expand Up @@ -673,7 +643,7 @@ def item_function_wrapper(*args, **kwargs):
if not self.results_always:
shutil.rmtree(result_dir)
for image_type in ['baseline_image', 'diff_image', 'result_image']:
summary[image_type] = None # image no longer %EXISTS%
summary[image_type] = None # image no longer exists
else:
self._test_results[str(pathify(test_name))] = summary
pytest.fail(msg, pytrace=False)
Expand Down Expand Up @@ -704,21 +674,6 @@ def pytest_unconfigure(self, config):
json.dump(self._generated_hash_library, fp, indent=2)

if self.generate_summary:
# Generate a list of test directories
dir_list = [p.relative_to(self.results_dir)
for p in self.results_dir.iterdir() if p.is_dir()]

# Resolve image paths
for directory in dir_list:
test_name = directory.parts[-1]
for image_type, filename in [
('baseline_image', 'baseline.png'),
('diff_image', 'result-failed-diff.png'),
('result_image', 'result.png'),
]:
if self._test_results[test_name][image_type] == '%EXISTS%':
self._test_results[test_name][image_type] = str(directory / filename)

if 'json' in self.generate_summary:
summary = self.generate_summary_json()
print(f"A JSON report can be found at: {summary}")
Expand Down
6 changes: 6 additions & 0 deletions pytest_mpl/summary/templates/filter.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@ <h5>Sort tests by...</h5>
{{ sort_option('status-sort', 'status', 'desc', default=true) }}
{{ sort_option('collected-sort', 'collected', 'asc') }}
{{ sort_option('test-name', 'name') }}
{% if results.warn_missing['baseline_image'] -%}
{{ sort_option('rms-sort', 'RMS', 'desc') }}
{%- endif %}
</div>
<form id="filterForm" onsubmit="return false;">
<h5>Show tests which have...</h5>
Expand All @@ -32,16 +34,20 @@ <h5>Show tests which have...</h5>
{{ filter_option('overall-failed', 'failed') }}
{{ filter_option('overall-skipped', 'skipped') }}
</div>
{% if results.warn_missing['baseline_image'] -%}
<div class="list-group m-2">
{{ filter_option('image-match', 'matching images') }}
{{ filter_option('image-diff', 'differing images') }}
{{ filter_option('image-missing', 'no baseline image') }}
</div>
{%- endif %}
{% if results.warn_missing['baseline_hash'] -%}
<div class="list-group m-2 mpl-hash">
{{ filter_option('hash-match', 'matching hashes') }}
{{ filter_option('hash-diff', 'differing hashes') }}
{{ filter_option('hash-missing', 'no baseline hash') }}
</div>
{%- endif %}
<div class="d-flex">
<button type="submit" class="btn btn-primary m-2" data-bs-dismiss="offcanvas">Apply</button>
<button type="submit" class="btn btn-outline-secondary m-2" onclick="resetFilters()">Reset</button>
Expand Down
Loading