Provide detailed steps of how you can execute tests on your local machine before creating a PR

[hostep]

Community engineering issue

Type of Doc:

• New topic
• Topic Update, Devdoc link:
• Other

Magento 2 version:

• 2.1.X
• 2.2.X
• 2.3.X
• back port
• forward port

Notifications of Changes:

• Changes to Admin interface?
• New software requirements?
• New REST endpoints or updated GraphQL query syntax?
• New command line option?

Content

Hi!

When creating a PR on the magento2 repo, there are a bunch of automated tests which get executed. But there currently doesn't seem to be a clear explanation of how to execute these tests yourself.
It would be nice if we have some documentation which says how to run those tests, so we can already prepare the PR ourselves to get accepted faster and not having to go back and forth and wait until the automated tests have run after changing something.
There is already some good documentation around how to run unit tests, but the other tests (static tests for example) are not documented properly I think. There is this page, which says to call bin/magento dev:tests:run static, but that doesn't seem to give the same results as the static tests ran through automatisation.

Additional information

For example, I created a PR magento/magento2#22733 which apparently failed on some static tests. And then a Magento developer fixed those. Which is great btw, but I would like to be able to see those failures myself and fix them myself if possible before creating the PR.
When asking how to execute these static test, I didn't get a very clear answer, so I went looking in the documentation where to find information regarding that and I couldn't find anything around it (unless I missed it somehow).

Thanks!

[lorikrell]

@hostep, great request! I'm working with @dmanners on a set of Beginner Guides and the wiki for Magento2 repo to document some of this information. I've assigned myself to the ticket, and will work towards that content.

I'll also make sure this is linked/noted in the Contributor Guide in DevDocs.

We've covered the Issue process. Next is PR! More info soon for this one.

[DanielRuf]

Exactly the issue that I face at the moment.
It is not clear how and what is running behind the curtains on the new private CI setup (no logs?).

[erfanimani]

I can second this request. The instructions on running tests are quite confusing at the moment.

Is it for core code? Is it for custom code in app/code? Are you supposed to use the phpcs/phpunit commands or dev:tests:run? What's up with Magento ignoring Magento2.Files.LineLength.MaxExceeded when the Magento2 standard actually throws Generic.Files.LineLength.TooLong? Why is the Magento2 coding standard failing on core code? Why is Squiz.ControlStructures.ControlSignature.SpaceAfterCloseParenthesis being thrown when running phpcs with --standard=Magento2 even though that's clearly against Magento's standards. And so on...