Magento Test Framework giving the failure results for Magento 2.1.0 ce

[shahankit783]

When I tried to run the default testcases of magento 2.1.0 ce using MTF , only few of the test cases are passing rest of all are failing. I had also changed the xml file configurations for the same

Preconditions

1. Windows 8
2. Magento Version: 2.1.0 ce
3. SQL Version: 5.7.11
4. PHP Version: 5.6.19

Test Run Steps:

1)Go to Magento Root Directory:
cd magento2.1\dev\tests\functional

2. Now run the test case execution by following command:
   vendor\bin\phpunit

3. I am also running the specific group by following command:
   **vendor\bin\phpunit --group "Bundle_Product_(CS)"

Expected result

Most of the test cases should "PASS" as per the Dev-docs.

Actual result

Failure Error:
Though there are many failure reasons but the very common Exception it shows is : "Configuration settings are not applied! Url: http://localhost:8080/magento2.1.0ce/admin_iaa328/admin/system_config/save/section/admin"

Please find the attached zip file in which i have added 3 configuration files (modified for running test cases)

Config_files.zip

Please provide the solution or the steps for configuration (if I missed any)

Thanks!!!!

[lanken]

Hi @shahankit783,

Could you, please, change your hostname from 'localhost' to something like 'localhost.com', cURL isn't not working correctly in this case.

Also, could you tell what is the version of your web browser and selenium server?

One more item, pay attention to the app_backend_url, it must be exact the same as the url after login to the Magento admin, if it contains index.php in it, it must be also specified in the phpunit.xml.

[shahankit783]

Hello,
I have changed host name from localhost to localhost.com and it is running on port 8080.

Chrome Browser: Version 52.0.2743.116 m (64-bit)
Selenium jar :- selenium-server-standalone-3.0.0-beta2.jar

app_backend_url :- http://localhost.com:8080/magento2.1.0ce/admin_iaa328/

[lanken]

Could you, please, use stable selenium server releases: http://selenium-release.storage.googleapis.com/index.html (for example, 2.53)
What is your URL after you have logged in to the admin manually?

[shahankit783]

I have tried to run it with selenium-server-standalone-2.53.1.jar with chrome browser.

The most common error in all failure test cases is "Exception: Configuration settings are not applied! Url: http://localhost.com:8080/magento2.1.0ce/admin_iaa328/admin/system_config/save/section/admin"

Manually URL which i fired in browser :-
BACK END URL :- http://localhost:8080/magento2.1.0ce/admin_iaa328/admin

after login it will redirected to below URL:
http://localhost:8080/magento2.1.0ce/admin_iaa328/admin/dashboard/

It might be possible that i am missing some of the configuration settings in magento 2.1.ce.

I have followed the configuration and settings from following URL only:
http://devdocs.magento.com/guides/v2.1/mtf/mtf_introduction.html

If magento needs to be configured manually for different modules, then can you please suggest which are the settings that needs to be apply before running MTF.

PS: I am novice in magento and i am working with MTF to verify the magento provided testcases.

[lanken]

Add '/' at the end of the backend url
http://localhost:8080/magento2.1.0ce/admin_iaa328/admin/

I guess, it's mistake, but the host name according to your words should be localhost.con, not localhost.

Also, have you set your credentials to log in to the admin?
http://devdocs.magento.com/guides/v2.1/mtf/mtf_quickstart/mtf_quickstart_config.html#mtf_quickstart_config_configxml

Could you, please, attach the curl response error. It should be in the folder dev/tests/functional/var/log/magento /module/test_case/variation/curl-response

[shahankit783]

Yes in phpunit.xml it is the same as you have mentioned .

app_backend_url ="http://localhost.com:8080/magento2.1.0ce/admin_iaa328/"
app_frontend_url ="http://localhost.com:8080/magento2.1.0ce/"

[lanken]

What is about curl response?
Also, please, check http://devdocs.magento.com/guides/v2.1/mtf/mtf_quickstart/mtf_quickstart_magento.html especially Admin Account Sharing

[shahankit783]

Yes, I have applied the same settings as mentioned in that guide.
When i try to run the specific group like "Cross-sells_(MX)" it does open the browser but it won't open any URL inside browser.

The following is stack of error:
D:\wamp64\www\magento2.1.0ce\dev\tests\functional>vendor\bin\phpunit --group Cross-sells_(MX)

PHPUnit 4.1.0 by Sebastian Bergmann.

Configuration read from D:\wamp64\www\magento2.1.0ce\dev\tests\functional\phpunit.xml

Time: 52.57 seconds, Memory: 55.75MB

There was 1 error:

1. Magento\Catalog\Test\TestCase\Product\AddToCartCrossSellTest::test with data set "AddToCartCrossSellTestVariation1"
   AddToCartCrossSellTestVariation1)
   Exception: Configuration settings are not applied! Url: http://localhost.com:8080/magento2.1.0ce/admin_iaa328/admin/sysem_config/save/section/admin

D:\wamp64\www\magento2.1.0ce\dev\tests\functional\tests\app\Magento\Config\Test\Handler\ConfigData\Curl.php:126
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\tests\app\Magento\Config\Test\Handler\ConfigData\Curl.php:42
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\Fixture\InjectableFixture.php:257
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\lib\Magento\Mtf\Util\Protocol\CurlTransport\WebapiDecorator.php:125
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\lib\Magento\Mtf\Util\Protocol\CurlTransport\WebapiDecorator.php:102
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\lib\Magento\Mtf\Util\Protocol\CurlTransport\WebapiDecorator.php:89
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\ObjectManager\Factory.php:245
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\ObjectManager\ObjectManager.php:53
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\ObjectManager\Factory.php:156
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\ObjectManager\Factory.php:236
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\ObjectManager\ObjectManager.php:53
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\Factory\AbstractFactory.php:48
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\Fixture\InjectableFixture.php:257
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\tests\app\Magento\Catalog\Test\Fixture\Product\CategoryIds.php:60
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\ObjectManager\Factory.php:245
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\ObjectManager\ObjectManager.php:53
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\Factory\AbstractFactory.php:48
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\Fixture\InjectableFixture.php:225
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\Fixture\InjectableFixture.php:162
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\ObjectManager\Factory.php:245
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\ObjectManager\ObjectManager.php:53
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\Factory\AbstractFactory.php:48
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\Fixture\FixtureFactory.php:51
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\tests\app\Magento\Catalog\Test\TestCase\Product\AbstractProductPromot
dProductsTest.php:107
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\tests\app\Magento\Catalog\Test\TestCase\Product\AddToCartCrossSellTes
.php:67
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\TestCase\Injectable.php:217
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\TestCase\Functional.php:288
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\TestCase\Functional.php:202
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\TestCase\Injectable.php:201
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\magento\mtf\Magento\Mtf\TestCase\Injectable.php:158
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\phpunit\phpunit\src\TextUI\Command.php:179
D:\wamp64\www\magento2.1.0ce\dev\tests\functional\vendor\phpunit\phpunit\src\TextUI\Command.php:132
?[37;41m ?[0m
?[37;41mFAILURES! ?[0m
?[37;41mTests: 1, Assertions: 0, Errors: 1.?[0m

[lanken]

It's not a curl response, it's stack trace response.
It should be in the folder like
dev/tests/functional/var/log/magento /Catalog/AddToCartCrossSellTest/AddToCartCrossSellTestVariation1/curl-response

[shahankit783]

Please see the attached file which shows Curl Response.
Curl_Response.txt

From the Curl Response as you can see it is trying to go to the "http://localhost:8080/magento2.1.0ce/pub/static/frontend/Magento/luma/en_US" url , but as it is not found it gives 404 error.

[lanken]

According to response to send request to "http://localhost:8080/magento2.1.0ce/" not to "http://localhost.com:8080/magento2.1.0ce/"

[shahankit783]

When i manually run the both URLs ("http://localhost:8080/magento2.1.0ce/" or "http://localhost.com:8080/magento2.1.0ce/") in browser it works same and fine.

But when you read the Curl_Response file it shows the base URL is"http://localhost:8080/magento2.1.0ce/pub/static/frontend/Magento/luma/en_US". so Until pub URL is working. but after that the URL is not working.

[lanken]

Have you reinstalled Magento after changing host name or updated Magento base URLs?