-
Notifications
You must be signed in to change notification settings - Fork 5.9k
fix: skip flaky update test #5097
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
✨ code-server docs for PR #5097 is ready! It will be updated on every commit.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will approve but IMO we should not skip tests
I agree! This is a temporary fix. We shouldn't add tests that are flaky but sometimes we don't realize until it's already merged. I will make sure this tracked and addressed in the next version/sprint! |
Ironically, another test in that suite failed 🤦🏼♂️ https://github.com/coder/code-server/runs/6012334860?check_suite_focus=true#step:12:13 Should I delete the entire suite? |
https://github.com/jest-community/eslint-plugin-jest also seems relevant to enforcing good testing rules (i.e. no skipping tests) |
Ah sorry I missed your response! The Slack integration has betrayed me and I am only now going through emails. This is a long-standing one. It has gone through at least one cycle of being disabled and re-enabled but no one has ever sat down to figure out why it actually flakes (this is part of the reason I am hesitant to disable it again because likely it will never be fixed otherwise). I think the appropriate response is to keep running the tests until they pass, as frustrating as that is, and make fixing the flake a priority. One thing we should do is add some output around the actual values being compared since just seeing |
Ahhh...Okay that makes sense. I'm going to close this for now. |
I've noticed this test fail multiple times. I think we should skip it until we set aside time to rewrite it.
Evidence: https://github.com/coder/code-server/runs/5999223188?check_suite_focus=true#step:12:13
Fixes N/A