Skip to content

Commit cd1bda5

Browse files
KevinBKozanKevinBKozan
committed
MQE-1579: Outline test isolation best practices
- Added Test isolation article
1 parent 6675ee5 commit cd1bda5

File tree

1 file changed

+104
-0
lines changed

1 file changed

+104
-0
lines changed

‎docs/guides/test-isolation.md

+104
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# Test Isolation
2+
3+
Since MFTF is a framework for testing a highly customizable and ever changing application, MFTF tests need to be properly isolated.
4+
5+
## What is test isolation?
6+
7+
Test isolation refers to a test that does not leave behind any data or configurations in the Magento instance.
8+
9+
An MFTF test is considered fully isolated if
10+
11+
1. It does not leave data behind.
12+
2. It does not leave Magento configured in a different state than when the test started.
13+
3. It does not affect a following test's outcome.
14+
15+
## Why is isolation important?
16+
17+
As mentioned above, isolation is important because poor isolation can lead to other test failures. For a test to be useful, you must have high confidence in the test's outcome, and by introducing test isolation issues it can invalidate a test's result.
18+
19+
## How can I achieve test isolation?
20+
21+
This is difficult to do given how large the Magento application is, but a systematic approach can ensure a high level of confidence in you test's isolation.
22+
23+
### Cleaning up data
24+
25+
If your test creates any data via `<createData>` then a subsequent `<deleteData>` action *must* exist in the test's `<after>` block.
26+
27+
This includes both `<createData>` actions in the test's `<before>` as well as in the test body.
28+
29+
```xml
30+
<test name="SampleTest">
31+
<before>
32+
<createData entity="SimpleSubCategory" stepKey="category"/>
33+
</before>
34+
<after>
35+
<deleteData createDataKey="category" stepKey="deleteCategory"/>
36+
<deleteData createDataKey="entityCreatedDuringWorkflow" stepKey="deleteCategory"/>
37+
</after>
38+
...
39+
<createData entity="SimpleSubCategory" stepKey="entityCreatedDuringWorkflow"/>
40+
...
41+
</test>
42+
```
43+
44+
Other test data can be more difficult to detect, and requires an understanding of what the test does in its workflow.
45+
46+
```xml
47+
<test name="AdminAddImageForCategoryTest">
48+
<before>
49+
<actionGroup ref="LoginAsAdmin" stepKey="loginAsAdmin"/>
50+
</before>
51+
<after>
52+
<actionGroup ref="DeleteCategory" stepKey="DeleteCategory">
53+
<argument name="categoryEntity" value="SimpleSubCategory"/>
54+
</actionGroup>
55+
<actionGroup ref="logout" stepKey="logout"/>
56+
</after>
57+
<!-- Go to create a new category with image -->
58+
<actionGroup ref="goToCreateCategoryPage" stepKey="goToCreateCategoryPage"/>
59+
...
60+
</test>
61+
```
62+
63+
Note that the test contains a context setting comment describing the workflow; this is very helpful in determining that a new category will be created, which will need to be cleaned up in the test `<after>` block.
64+
65+
### Cleaning up configuration
66+
67+
Similarly, configuration changes can be easily identified by `<magentoCLI>` actions.
68+
69+
```xml
70+
<test name="AddOutOfStockProductToCompareListTest">
71+
<before>
72+
<magentoCLI command="config:set cataloginventory/options/show_out_of_stock 0" stepKey="displayOutOfStockNo"/>
73+
...
74+
</before>
75+
<after>
76+
<magentoCLI command="config:set cataloginventory/options/show_out_of_stock 1" stepKey="displayOutOfStockNo"/>
77+
...
78+
</after>
79+
...
80+
</test>
81+
```
82+
83+
Configuration changes can also be done via `<createData>` actions, but that is not recommended as it is much easier to identify `<magentoCLI>` commands.
84+
85+
A test's workflow can also alter the application's configuration, and much like data cleanup, this can only be identified by understanding a test's workflow:
86+
87+
```xml
88+
<test name="AdminMoveProductBetweenCategoriesTest">
89+
...
90+
<!-- Enable `Use Categories Path for Product URLs` on Stores -> Configuration -> Catalog -> Catalog -> Search Engine Optimization -->
91+
<amOnPage url="{{AdminCatalogSearchConfigurationPage.url}}" stepKey="onConfigPage"/>
92+
<waitForPageLoad stepKey="waitForLoading"/>
93+
<conditionalClick selector="{{AdminCatalogSearchEngineConfigurationSection.searchEngineOptimization}}" dependentSelector="{{AdminCatalogSearchEngineConfigurationSection.openedEngineOptimization}}" visible="false" stepKey="clickEngineOptimization"/>
94+
<uncheckOption selector="{{AdminCatalogSearchEngineConfigurationSection.systemValueUseCategoriesPath}}" stepKey="uncheckDefault"/>
95+
<selectOption userInput="Yes" selector="{{AdminCatalogSearchEngineConfigurationSection.selectUseCategoriesPatForProductUrls}}" stepKey="selectYes"/>
96+
<click selector="{{AdminConfigSection.saveButton}}" stepKey="saveConfig"/>
97+
<waitForPageLoad stepKey="waitForSaving"/>
98+
...
99+
</test>
100+
```
101+
102+
One thing to note, unless a test is specifically testing the configuration page's frontend capabilities, configuring the application should always be done with a `<magentoCLI>` action.
103+
104+
<!-- Link definitions -->

0 commit comments

Comments
 (0)