fix: detect newly added specs in dev-server compilation #17950

ZachJW34 · 2021-08-31T21:09:51Z

Closes Must restart Cypress to run newly added spec file when using Webpack 5 #16664

User facing changelog

Additional details

There are more fine-grained API's for controlling loaders. I tried a few, such as addDependency and addContextDependency without much luck. This loader is pretty cheap so I don't think disabling the caching would have much of a performance hit. Also, I still need to write a test covering this case.

How has the user experience changed?

PR Tasks

Have tests been added/updated?
Has the original issue or this PR been tagged with a release in ZenHub?
Has a PR for user-facing changes been opened in cypress-documentation?
Have API changes been updated in the type definitions?
Have new configuration options been added to the cypress.schema.json?

cypress-bot · 2021-08-31T21:09:54Z

Thanks for taking the time to open a PR!

Create a Draft Pull Request if your PR is not ready for review. Mark the PR as Ready for Review when you're ready for a Cypress team member to review the PR.
Become familiar with the Code Review Checklist for guidelines on coding standards and what needs to be done before a PR can be merged.

cypress · 2021-08-31T21:17:59Z

Test summary

19173 • 0 • 218 • 0 • 14

Run details


Project	cypress
Status	Passed
Commit	`5a1a37a`
Started	Feb 8, 2022 10:44 PM
Ended	Feb 8, 2022 10:56 PM
Duration	11:33 💡
OS	Linux Debian - 10.10
Browser	Multiple

View run in Cypress Dashboard ➡️

Flakiness

	commands/net_stubbing_spec.ts	2
1	network stubbing > waiting and aliasing > can timeout waiting on a single request using "alias.request"
2	network stubbing > waiting and aliasing > can timeout waiting on a single request using "alias.request"
	e2e/e2e_cookies_spec.js	3
1	e2e cookies spec > __Host- prefix > can set __Host- cookie
2	e2e cookies spec > __Host- prefix > errors when __Host- cookie and secure:false
3	e2e cookies spec > __Host- prefix > errors when __Host- cookie and path
This comment includes only the first 5 flaky tests. See all 14 flaky tests in the Cypress Dashboard.

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

JessicaSachs · 2021-09-01T02:25:03Z

npm/webpack-dev-server/src/loader.ts

@@ -49,7 +50,11 @@ function buildSpecs (projectRoot: string, files: Cypress.Cypress['spec'][] = [])
 }

 // Runs the tests inside the iframe
-export default function loader (this: CypressCTWebpackContext) {
+export default function loader (this: CypressCTWebpackContext & LoaderContext<any>) {


Don't you want unknown? CC @lmiller1990

I think we want <void>, we are intentionally not passing any additional options to the loader context.

I am not entirely clear on how this generic is used, but you rarely, if ever, want any (unless it's truly valid that your function can take any argument, which is pretty unusual).

lmiller1990 · 2021-09-01T08:49:47Z

This test might be close to what we need: https://github.com/cypress-io/cypress/blob/develop/npm/webpack-dev-server/test/e2e.spec.ts#L131

Basically when we make a new file, we want to ensure that webpack re-runs and emits the right event - that looks like what this test is doing.

I tested this and it works great! I notice this message:

"Not cacheable". I wonder if there's any perf implications.

lmiller1990 · 2021-09-01T09:07:21Z

npm/webpack-dev-server/src/loader.ts

+  // In Webpack 5, a spec added after the dev-server is created won't
+  // be included in the compilation. Disabling the caching of this loader ensures
+  // we regenerate our specs and include any new ones in the compilation.
+  this.cacheable(false)


Do we want this to be conditional (if we are using Webpack 5)? Does this.cacheable even exist in Webpack 4?

It won't throw even if it doesn't. I'm fine with this.

this.cacheable is valid for Webpack 4 and 5
https://v4.webpack.js.org/api/loaders/#thiscacheable

ZachJW34 · 2021-09-15T21:59:18Z

@lmiller1990 I was able to get some performance metrics after looking into #17920. Here is what I found testing inside a Next repo using Webpack 5:

Cache Enabled
  - Cold Start (ms): [2584, 262, 275, 257, 279]
  - Code Change (ms): [55, 58, 62, 41, 44]
 
Cache Disabled ("cache: false" in webpack config):
  - Cold Start (ms): [2747, 2541, 2453, 2214, 2591]
  - Code Change (ms): [1140, 729, 640, 591, 528]
 
Loader Cache Disabled ("this.cacheable(false)" in loader):
  - Cold Start (ms): [2679, 261, 290, 300, 274]
  - Code Change (ms): [53, 58, 58, 62, 50]

"Cold start" was the time it took for Webpack to when running npx cypress open-ct and the "Code Change" was the time it took Webpack to recompile after changing a spec file

Adding the cache: false option to the webpack.config.js fixes the spec detection issue but at a high performance cost.
Conclusion: The current implementation (disabling the caching of the loader) has very little performance impact.

lmiller1990 · 2021-09-20T02:16:01Z

Great! Still in draft - any outstanding changes @ZachJW34 and can we merge?

Zacharias3690 · 2021-09-20T02:43:03Z

@lmiller1990 Looks like you tagged the wrong Zach :) Think you were looking for @ZachJW34

lmiller1990 · 2021-09-20T05:31:19Z

@Zacharias3690 updated tagging the correct person... sorry about that! 🤦

…-detect-new-spec

ZachJW34 · 2022-02-07T21:03:47Z

@lmiller1990 fixed the tests by adding --ignore-scripts to the yarn installs for the npm specific tests. Should be good to merge now

@ts-nocheck

* develop: feat: gray out the path to system node in cypress run header (#20121) feat: redesign server errors (#20072) test: fix awesome-typescript-loader test and remove test-binary job (#20131) fix: Fix issues with stack traces and command log in Chrome 99 (#20049) fix: `cy.type(' ')` fires click event on button-like elements. (#20067) fix: `change`, `input` events are not fired when the same option is selected again. (#19623) build: publish vue3 on latest (#20099) chore: release @cypress/webpack-preprocessor-v5.11.1 chore: release @cypress/webpack-dev-server-v1.8.1 fix: detect newly added specs in dev-server compilation (#17950) chore: Remove pkg/driver //@ts-nocheck part 3 (#19837) chore: set up semantic-pull-request GitHub Action (#20091) chore: release @cypress/react-v5.12.2 fix: remove nullish coalescing in js files to support node 12 (#20094) docs: update @cypress/webpack-preprocessor links (#19902) refactor: use aliases instead of meta (#19566)

ZachJW34 requested a review from lmiller1990 August 31, 2021 21:09

JessicaSachs reviewed Sep 1, 2021

View reviewed changes

lmiller1990 reviewed Sep 1, 2021

View reviewed changes

lmiller1990 mentioned this pull request Sep 8, 2021

[Component Testing] New test code is not being bundled using NextJS 11.1 and Webpack 5 #17920

Closed

fix: detect newly added specs in dev-server compilation

8a1e6d4

ZachJW34 force-pushed the issue-16664-cypress-detect-new-spec branch from 1f20c0b to 8a1e6d4 Compare September 17, 2021 19:58

Merge branch 'master' into issue-16664-cypress-detect-new-spec

0bd85df

lmiller1990 previously approved these changes Sep 20, 2021

View reviewed changes

ZachJW34 marked this pull request as ready for review September 20, 2021 02:15

lmiller1990 mentioned this pull request Jan 19, 2022

No tests found, When rename *.ts to *.spec.ts for component testing #19717

Closed

Merge remote-tracking branch 'origin/master' into issue-16664-cypress…

a003154

…-detect-new-spec

emilyrohrbough mentioned this pull request Jan 25, 2022

Investigate Webpack 5 Impact #19555

Closed

ZachJW34 added 2 commits February 7, 2022 14:14

Merge remote-tracking branch 'origin/master' into issue-16664-cypress…

bf61cb3

…-detect-new-spec

ingore-scripts

5351ef2

ZachJW34 dismissed lmiller1990’s stale review via 5351ef2 February 7, 2022 20:46

lmiller1990 self-requested a review February 8, 2022 21:42

lmiller1990 approved these changes Feb 8, 2022

View reviewed changes

JessicaSachs approved these changes Feb 8, 2022

View reviewed changes

Merge branch 'master' into issue-16664-cypress-detect-new-spec

5a1a37a

ZachJW34 merged commit f9ce67c into master Feb 8, 2022

ZachJW34 deleted the issue-16664-cypress-detect-new-spec branch February 8, 2022 23:06

cypress-bot bot mentioned this pull request Feb 8, 2022

Must restart Cypress to run newly added spec file when using Webpack 5 #16664

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: detect newly added specs in dev-server compilation #17950

fix: detect newly added specs in dev-server compilation #17950

Uh oh!

ZachJW34 commented Aug 31, 2021 •

edited

Loading

Uh oh!

cypress-bot bot commented Aug 31, 2021

Uh oh!

cypress bot commented Aug 31, 2021 •

edited

Loading

Uh oh!

JessicaSachs Sep 1, 2021

Uh oh!

lmiller1990 Sep 1, 2021 •

edited

Loading

Uh oh!

lmiller1990 commented Sep 1, 2021 •

edited

Loading

Uh oh!

lmiller1990 Sep 1, 2021

Uh oh!

JessicaSachs Sep 7, 2021

Uh oh!

ZachJW34 Sep 17, 2021

Uh oh!

ZachJW34 commented Sep 15, 2021

Uh oh!

lmiller1990 commented Sep 20, 2021 •

edited

Loading

Uh oh!

Zacharias3690 commented Sep 20, 2021

Uh oh!

lmiller1990 commented Sep 20, 2021 •

edited

Loading

Uh oh!

ZachJW34 commented Feb 7, 2022

Uh oh!

Uh oh!

fix: detect newly added specs in dev-server compilation #17950

fix: detect newly added specs in dev-server compilation #17950

Uh oh!

Conversation

ZachJW34 commented Aug 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

User facing changelog

Additional details

How has the user experience changed?

PR Tasks

Uh oh!

cypress-bot bot commented Aug 31, 2021

Uh oh!

cypress bot commented Aug 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test summary

Run details

Flakiness

Uh oh!

JessicaSachs Sep 1, 2021

Choose a reason for hiding this comment

Uh oh!

lmiller1990 Sep 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lmiller1990 commented Sep 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lmiller1990 Sep 1, 2021

Choose a reason for hiding this comment

Uh oh!

JessicaSachs Sep 7, 2021

Choose a reason for hiding this comment

Uh oh!

ZachJW34 Sep 17, 2021

Choose a reason for hiding this comment

Uh oh!

ZachJW34 commented Sep 15, 2021

Uh oh!

lmiller1990 commented Sep 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Zacharias3690 commented Sep 20, 2021

Uh oh!

lmiller1990 commented Sep 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ZachJW34 commented Feb 7, 2022

Uh oh!

Uh oh!

ZachJW34 commented Aug 31, 2021 •

edited

Loading

cypress bot commented Aug 31, 2021 •

edited

Loading

lmiller1990 Sep 1, 2021 •

edited

Loading

lmiller1990 commented Sep 1, 2021 •

edited

Loading

lmiller1990 commented Sep 20, 2021 •

edited

Loading

lmiller1990 commented Sep 20, 2021 •

edited

Loading