Generating tests fails for all if a single test fails

[magicsunday]

Running command vendor/bin/mftf generate:tests gets error.

Preconditions

Magento version 2.3.0-beta32
Mftf version 2.3.11

Steps to reproduce

Run `vendor/bin/mftf generate:tests`

Expected result

The tests should been generated.

Actual result

Currently due an error in a third party extension (dotmailer/dotmailer-magento-extension#306) the command fails to generate the tests. This shouldn't happen imho.

If I manually disable the exception thrown in vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Test/Util/ActionMergeUtil.php (Line 278) the command runs fine and all the other tests are generated.

Imho the command shouldn't fail if any test fails to be generated. I should instead print a warning or log the error and proceed with the next test.

[tomreece]

Internal ticket
https://jira.corp.magento.com/browse/MQE-1800

@magicsunday Thank you for your issue even though it has been almost a year now. Sorry about that. We are improving our interactions with Github.

I am interested in this idea. In the past we've had reasons to fail-fast, but we need to re-evaluate those reasons. I have created an internal ticket and we discuss further.

[lbajsarowicz]

@tomreece and @okolesnyk

Could you provide the final expectations according to "fail-fast"?
I can address the issue, but need to have confirmed what is the final version?

[tomreece]

The problem is if some tests aren't generated then they won't be run in our CICD environment. This would be bad.

I think we should keep MFTF the same, fail to generate if any test is bad.

[vasylmalanka]

@tomreece it would be good to have an option for this (e.g. --skip-failed or --ignore-failing)

[okolesnyk]

@lbajsarowicz @tomreece @vasylmalanka
I think we will end up generating all tests and put into logs a list of tests which had issues.
We also should introduce static check to be able to fail build when something is not generating.

Also I believe we should output error when we execute generate test for specific test. It will simplify development process.

[tomreece]

We discussed this today in grooming and we could not agree on how to implement it. We need your feedback.

Context
Someone else's tests that fail to generate, such as those in another module, should not stop my tests from generating. The idea is, if we have 1000 tests and 2 tests have mistakes then we will generate 998 tests and output a log message about the other 2 tests instead of failing the entire process like we do today.

But we can't decide between:

a) Add a flag, like generate:tests --ignore-failing
b) Make it the default behavior of generate:tests
c) generate:tests will return a nonzero exit code like today, but still generate every test that it could

Option A -- add a flag

• Pros
  • Does not require any changes to our internal CICD pipeline
• Cons
  • We want to avoid adding lots of flags to our commands

Option B -- make it the default

• Pros
  • More intuitive that it just works
• Cons
  • We have to change our internal CICD to fail the build if tests don't generate. This can be a lot of work.

Option C -- nonzero exit code, but still generate all else

• Pros
  • Maintains current functionality of nonzero exit code
  • But still generates everything else
  • Best of both worlds
• Cons
  • None that I can think of.

What do you think?

For option A put a 👍
For option B put a ❤️
For option C put a 🚀

[navarr]

There was an idea posted on twitter that I quite liked, and that I think would pair nicely with all of the above:

When a test cannot be generated, generate a stub for the test that fails - detailing the generation error.

[jilu1]

The internal ticket https://jira.corp.magento.com/browse/MQE-1800 is ready for development for next release which is targeted toward the end of August. @lbajsarowicz will you be able to work on this issue with this schedule? Please let us know. Thanks!

[jilu1]

AC

• Generate:tests generates every *Cest file that it can
• It does not stop when it encounters a bad test. Log testnames and return number of failed test generations in the console
  "X tests failed to generate, please see for details"
• If any tests failed to generate, return non-zero exit code. Build should still fail if any tests fail to generate.
• Unit Tests

[jilu1]

@lbajsarowicz Could you please let us know by end of this week if you will be able to work on this or not? Thanks!