Skip to content

fix: skip flaky update test #5097

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

fix: skip flaky update test #5097

wants to merge 1 commit into from

Conversation

jsjoeio
Copy link
Contributor

@jsjoeio jsjoeio commented Apr 13, 2022

I've noticed this test fail multiple times. I think we should skip it until we set aside time to rewrite it.

Evidence: https://github.com/coder/code-server/runs/5999223188?check_suite_focus=true#step:12:13

Fixes N/A

@jsjoeio jsjoeio added the testing Anything related to testing label Apr 13, 2022
@jsjoeio jsjoeio added this to the April 2022 milestone Apr 13, 2022
@jsjoeio jsjoeio self-assigned this Apr 13, 2022
@jsjoeio jsjoeio requested a review from a team April 13, 2022 17:48
@github-actions
Copy link

✨ code-server docs for PR #5097 is ready! It will be updated on every commit.

Copy link
Member

@code-asher code-asher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will approve but IMO we should not skip tests

@jsjoeio
Copy link
Contributor Author

jsjoeio commented Apr 13, 2022

Will approve but IMO we should not skip tests

I agree! This is a temporary fix. We shouldn't add tests that are flaky but sometimes we don't realize until it's already merged. I will make sure this tracked and addressed in the next version/sprint!

@jsjoeio
Copy link
Contributor Author

jsjoeio commented Apr 13, 2022

Ironically, another test in that suite failed 🤦🏼‍♂️

https://github.com/coder/code-server/runs/6012334860?check_suite_focus=true#step:12:13

Should I delete the entire suite?

@jsjoeio
Copy link
Contributor Author

jsjoeio commented Apr 13, 2022

https://github.com/jest-community/eslint-plugin-jest also seems relevant to enforcing good testing rules (i.e. no skipping tests)

@code-asher
Copy link
Member

code-asher commented Apr 18, 2022

Ah sorry I missed your response! The Slack integration has betrayed me and I am only now going through emails.

This is a long-standing one. It has gone through at least one cycle of being disabled and re-enabled but no one has ever sat down to figure out why it actually flakes (this is part of the reason I am hesitant to disable it again because likely it will never be fixed otherwise).

I think the appropriate response is to keep running the tests until they pass, as frustrating as that is, and make fixing the flake a priority.

One thing we should do is add some output around the actual values being compared since just seeing true and false is not very helpful in determining why the test is failing.

@jsjoeio
Copy link
Contributor Author

jsjoeio commented Apr 19, 2022

Ahhh...Okay that makes sense. I'm going to close this for now.

@jsjoeio jsjoeio closed this Apr 19, 2022
@jsjoeio jsjoeio deleted the jsjoeio/fix-flaky-test branch April 19, 2022 18:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
testing Anything related to testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants