Skip to content

chore(tests): Add initial full-journey tests #569

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 44 commits into from
Jun 28, 2023
Merged

chore(tests): Add initial full-journey tests #569

merged 44 commits into from
Jun 28, 2023

Conversation

jackbrewer
Copy link
Contributor

@jackbrewer jackbrewer commented Apr 5, 2023

This PR adds a lot of setup to enable e2e-style testing – including validating log lines written, data sent to backend, summaries surfaced in UI via plugin events.

It also includes a few initial tests to get a feel for the format. Currently they are one journey-type per file. Covered journeys are:

  • Completed plugin run – success (calls show onPostBuild)
  • Completed plugin run – success (calls show onSuccess)
  • Completed plugin run – path not found (calls show onPostBuild)
  • Completed plugin run – path not found (calls show onSuccess)
  • Failed plugin run – thresholds not met (calls failBuild onPostBuild)
  • Failed plugin run – thresholds not met (calls failPlugin onSuccess)
  • Settings – custom locale
  • Settings – custom device preset

There is a lot of component mocking in place for these tests – Lighthouse, Chrome, Puppeteer. This reduced the test time from ~12s per test to ~12ms. We also mock console (log and error) to capture the content being output to build logs, but this can cause problems during testing as debugging logs get swallowed up too! Maybe in future we use a custom logging function which calls console.log/error or a mock function depending on live/test environment.

99% of the changes here are test-related files, within the e2e folder. Very little has changes in the plugin itself (a file name change, and some removed colors).

@netlify
Copy link

netlify bot commented Apr 5, 2023

👷 Deploy Preview for plugin-lighthouse processing.

Name Link
🔨 Latest commit 4857cb3
🔍 Latest deploy log https://app.netlify.com/sites/plugin-lighthouse/deploys/649c4c16a0250f0008a66284

@@ -8,7 +8,7 @@
"local-onsuccess": "RUN_ON_SUCCESS=true node -e 'import(\"./src/index.js\").then(index => index.default()).then(events => events.onSuccess());'",
"lint": "eslint 'src/**/*.js'",
"format": "prettier --write 'src/**/*.js'",
"test": "node --experimental-vm-modules node_modules/jest/bin/jest.js --collect-coverage",
"test": "node --experimental-vm-modules node_modules/jest/bin/jest.js --collect-coverage --maxWorkers=1",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're currently not mocking express, so running multiple tests in parallel means we get domain conflicts. Adding this option runs tests sequentially (~4s total time at the moment). This could be a good follow-up issue to look into

@@ -53,7 +53,8 @@
"husky": "^8.0.1",
"jest": "^29.0.0",
"netlify-plugin-cypress": "^2.2.1",
"prettier": "^2.0.0"
"prettier": "^2.0.0",
"strip-ansi": "^7.0.1"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added to be able to more clearly compare log lines containing chalk colouring without needing to add the weird escape characters in our fixtures

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of these test runs follow a very similar format, other than for "success" based tests, I only mock console.error when testing that specifically, whereas "failure" based tests always need it mocked as we expect every test to throw a console error.

@@ -1,5 +1,3 @@
import chalk from 'chalk';
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed chalk from this file, as the generated strings are sent to the UI to be displayed – where we don't show color.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lighthouse.js was renamed to run-lighthouse.js to avoid confusion when mocking 'lighthouse'.

@jackbrewer jackbrewer requested a review from aitchiss April 5, 2023 16:20
@jackbrewer jackbrewer marked this pull request as ready for review April 5, 2023 16:20
@jackbrewer jackbrewer requested a review from a team April 5, 2023 16:20
aitchiss
aitchiss previously approved these changes Apr 6, 2023
Copy link
Contributor

@aitchiss aitchiss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is amaaaaaaazing 😍 So excited to have this level of testing! Love how clearly organised the mocks are ✨

@aitchiss aitchiss changed the title Add initial full-journey tests chore(tests): Add initial full-journey tests Jun 28, 2023
@aitchiss aitchiss merged commit 07afa56 into main Jun 28, 2023
@aitchiss aitchiss deleted the jbr/e2eMocks branch June 28, 2023 15:08
@aitchiss aitchiss mentioned this pull request Jun 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants