Prevent flushing same tags several times #29885

siimm · 2020-09-03T12:20:32Z

Description (*)

If a product is saved in admin - changed name, price, stock, weight, added category to product. Then after indexing, the same tags are flushed several times.
Here is an example of default Magento behaviour on cleaning cache tags during indexing (different arrays belong to different indexers cleaning the cache):

root@c119d6a06fcb:/var/www/m2clean$  bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1

Array
(
)


Array
(
)


Array
(
    [0] => cat_p_1
    [1] => cat_c_2
    [2] => cat_c_1
    [3] => cat_c_2
    [4] => cat_c_3
    [5] => cat_c_1
)

Array
(
    [0] => cat_p_1
    [1] => cat_c_2
    [2] => cat_c_1
    [3] => cat_c_2
    [4] => cat_c_3
    [5] => cat_c_1
)


Array
(
    [0] => cat_p_1
    [1] => cat_c_2
    [2] => cat_c_1
    [3] => cat_c_2
    [4] => cat_c_3
    [5] => cat_c_1
)


Array
(
    [0] => cat_p_1
)

Array
(
    [0] => cat_p_1
    [1] => cat_p_1
    [2] => cat_p_1
    [3] => cat_p_1
    [4] => cat_c_2
    [5] => cat_c_1
    [6] => cat_c_2
    [7] => cat_c_3
    [8] => cat_c_1
)


Array
(
    [0] => cat_p_1
    [1] => cat_p_1
    [2] => cat_p_1
    [3] => cat_p_1
    [4] => cat_c_2
    [5] => cat_c_1
    [6] => cat_c_2
    [7] => cat_c_3
    [8] => cat_c_1
)

Array
(
    [0] => cat_p_1
    [1] => cat_p_1
    [2] => cat_p_1
    [3] => cat_p_1
    [4] => cat_p_1
    [5] => cat_c_2
    [6] => cat_c_1
    [7] => cat_c_2
    [8] => cat_c_3
    [9] => cat_c_1
)


Array
(
    [0] => cat_p_1
    [1] => cat_p_1
    [2] => cat_p_1
    [3] => cat_p_1
    [4] => cat_p_1
    [5] => cat_c_2
    [6] => cat_c_1
    [7] => cat_c_2
    [8] => cat_c_3
    [9] => cat_c_1
)

Array
(
    [0] => cat_p_1
    [1] => cat_p_1
    [2] => cat_p_1
    [3] => cat_p_1
    [4] => cat_p_1
    [5] => cat_p_1
    [6] => cat_c_2
    [7] => cat_c_1
    [8] => cat_c_2
    [9] => cat_c_3
    [10] => cat_c_1
)


Array
(
    [0] => cat_p_1
    [1] => cat_p_1
    [2] => cat_p_1
    [3] => cat_p_1
    [4] => cat_p_1
    [5] => cat_p_1
    [6] => cat_c_2
    [7] => cat_c_1
    [8] => cat_c_2
    [9] => cat_c_3
    [10] => cat_c_1
)


Array
(
    [0] => cat_p_1
    [1] => cat_p_1
    [2] => cat_p_1
    [3] => cat_p_1
    [4] => cat_p_1
    [5] => cat_p_1
    [6] => cat_c_2
    [7] => cat_c_1
    [8] => cat_c_2
    [9] => cat_c_3
    [10] => cat_c_1
)


Array
(
    [0] => cat_p_1
    [1] => cat_p_1
    [2] => cat_p_1
    [3] => cat_p_1
    [4] => cat_p_1
    [5] => cat_p_1
    [6] => cat_c_2
    [7] => cat_c_1
    [8] => cat_c_2
    [9] => cat_c_3
    [10] => cat_c_1
)


Array
(
    [0] => cat_p_1
    [1] => cat_p_1
    [2] => cat_p_1
    [3] => cat_p_1
    [4] => cat_p_1
    [5] => cat_p_1
    [6] => cat_c_2
    [7] => cat_c_1
    [8] => cat_c_2
    [9] => cat_c_3
    [10] => cat_c_1
)

Here is the same output with the changes applied:

root@c119d6a06fcb:/var/www/m2clean$  bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1
Array
(
)


Array
(
)


Array
(
    [0] => cat_p_1
    [1] => cat_c_2
    [2] => cat_c_1
    [4] => cat_c_3
)

Array
(
    [0] => cat_p_1
    [1] => cat_c_2
    [2] => cat_c_1
    [4] => cat_c_3
)


Array
(
    [0] => cat_p_1
    [1] => cat_c_2
    [2] => cat_c_1
    [4] => cat_c_3
)


Array
(
    [0] => cat_p_1
)

Array
(
    [0] => cat_p_1
    [4] => cat_c_2
    [5] => cat_c_1
    [7] => cat_c_3
)


Array
(
    [0] => cat_p_1
    [4] => cat_c_2
    [5] => cat_c_1
    [7] => cat_c_3
)

Array
(
    [0] => cat_p_1
    [5] => cat_c_2
    [6] => cat_c_1
    [8] => cat_c_3
)


Array
(
    [0] => cat_p_1
    [5] => cat_c_2
    [6] => cat_c_1
    [8] => cat_c_3
)

Array
(
    [0] => cat_p_1
    [6] => cat_c_2
    [7] => cat_c_1
    [9] => cat_c_3
)

Array
(
    [0] => cat_p_1
    [6] => cat_c_2
    [7] => cat_c_1
    [9] => cat_c_3
)

Array
(
    [0] => cat_p_1
    [6] => cat_c_2
    [7] => cat_c_1
    [9] => cat_c_3
)

Array
(
    [0] => cat_p_1
    [6] => cat_c_2
    [7] => cat_c_1
    [9] => cat_c_3
)


Array
(
    [0] => cat_p_1
    [6] => cat_c_2
    [7] => cat_c_1
    [9] => cat_c_3
)

Related Pull Requests

Fixed Issues (if relevant)

Fixes magento/magento2#<issue_number>

Manual testing scenarios (*)

Observe the output of \Magento\Framework\Indexer\CacheContext::getIdentities during indexer run, same tags are flushed up to 10 times per one indexer instance
...

Questions or comments

Together with #29699 it will probably reduce the amount of cache flushing significatly

Contribution checklist (*)

Pull request has a meaningful description of its purpose
All commits are accompanied by meaningful commit messages
All new or changed code is covered with unit/integration tests (if applicable)
All automated tests passed successfully (all builds are green)

Resolved issues:

resolves [Issue] Prevent flushing same tags several times #29890: Prevent flushing same tags several times

m2-assistant · 2020-09-03T12:20:35Z

Hi @siimm. Thank you for your contribution
Here is some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

@magento give me test instance - deploy test instance based on PR changes
@magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

@magento run all tests - run or re-run all required tests against the PR changes
@magento run <test-build(s)> - run or re-run specific test build(s)
For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names. Allowed build names are:

Database Compare
Functional Tests CE
Functional Tests EE,
Functional Tests B2B
Integration Tests
Magento Health Index
Sample Data Tests CE
Sample Data Tests EE
Sample Data Tests B2B
Static Tests
Unit Tests
WebAPI Tests

You can find more information about the builds here

ℹ️ Please run only needed test builds instead of all when developing. Please run all test builds before sending your PR for review.

For more details, please, review the Magento Contributor Guide documentation.

⚠️ According to the Magento Contribution requirements, all Pull Requests must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of Pull Requests happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

sidolov · 2020-09-03T14:19:09Z

@magento create issue

rogyar · 2020-09-03T19:18:20Z

Hi @siimm. Thank you for your collaboration. Nice catch!
Could I ask you to cover your change with a simple unit test, please? The unit test will assert that the method returns only unique values.

Thank you!

siimm · 2020-09-04T08:32:09Z

Hi @rogyar, attached a minimalistic test case to this PR
f17148f

lenaorobei · 2020-09-04T16:16:54Z

@magento run all tests

lib/internal/Magento/Framework/Indexer/Test/Unit/CacheContextTest.php

lib/internal/Magento/Framework/Indexer/CacheContext.php

gabrieldagama · 2020-09-07T14:46:39Z

The risk was set tomedium due to: the suggested changes may affect other areas/modules.

lenaorobei · 2020-09-08T14:29:20Z

@magento run all tests

ihor-sviziev · 2020-09-17T12:16:30Z

@magento run all tests

engcom-Bravo

❌ QA Failed

Hello @siimm
Looks like the changes provided in this PR do not solve the original issue. I'm getting the same Output before and after switching to the PR

Please, see the Manual testing scenario

Go to System - Index Management
Select All Indexers and change the Actions to Update by Schedule
Go to Catalog - Categories, and change the Category Name for at least two Categories, for example
Go to Catalog - Products and change the Price, Quantity, Weight for at least two Products
Run bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1
Observe the output of \Magento\Framework\Indexer\CacheContext::getIdentities

Pull Request state was updated. Re-review required.

siimm · 2020-10-05T06:33:25Z

@engcom-Bravo
Did you check the values what the getIdentities method returns? - from the screenshot I can assume the values were inspected inside of the method itself - the $ids variable is hinting that.

engcom-Bravo · 2020-10-05T14:32:24Z

✔️ QA Passed

Further investigation discovered that the changes provided in this PR, fix the original issue

Manual Testing scenario

Go to System - Index Management
Select All Indexers and change the Actions to Update by Schedule
Go to Catalog - Categories, and change the Category Name for at least two Categories, for example
Go to Catalog - Products and change the Price, Quantity, Weight for at least two Products
Run bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1
Observe the return of \Magento\Framework\Indexer\CacheContext::getIdentities

BEFORE applying changes

AFTER switching to the PR

engcom-Bravo · 2020-10-06T07:48:37Z

@rogyar @BarnyShergold Can you please review the PR again?

Thank you

rogyar · 2020-10-06T12:55:28Z

lib/internal/Magento/Framework/Indexer/Test/Unit/CacheContextTest.php

+     * @param array $expected
+     * @dataProvider getTagsDataProvider
+     */
+    public function testUniqueTags($tagsData, $expected)


Just a recommendation for the future to use type hinting whenever it's possible. It's a very useful habit in PHP 7.x

magento-engcom-team · 2020-10-06T12:56:33Z

Hi @rogyar, thank you for the review.
ENGCOM-8237 has been created to process this Pull Request

m2-assistant · 2020-10-07T23:18:07Z

Hi @siimm, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.

ensure tags uniqueness

256cedd

ghost added the Progress: pending review label Sep 3, 2020

magento-engcom-team added Component: Indexer Release Line: 2.4 Partner: Vaimo Pull Request is created by partner Vaimo partners-contribution Pull Request is created by Magento Partner labels Sep 3, 2020

sidolov added Priority: P3 May be fixed according to the position in the backlog. Severity: S3 Affects non-critical data or functionality and does not force users to employ a workaround. labels Sep 3, 2020

m2-assistant bot mentioned this pull request Sep 3, 2020

[Issue] Prevent flushing same tags several times #29890

Closed

rogyar self-assigned this Sep 3, 2020

add simple unit test for magento#29890

f17148f

BarnyShergold requested changes Sep 6, 2020

View reviewed changes

lib/internal/Magento/Framework/Indexer/Test/Unit/CacheContextTest.php Outdated Show resolved Hide resolved

lib/internal/Magento/Framework/Indexer/CacheContext.php Outdated Show resolved Hide resolved

ghost assigned BarnyShergold Sep 6, 2020

ghost added Progress: needs update and removed Progress: pending review labels Sep 6, 2020

fix typo, remove empty line in test

47761ec

gabrieldagama added Risk: low Risk: medium and removed Risk: low labels Sep 7, 2020

ihor-sviziev mentioned this pull request Sep 17, 2020

Cleaning caches send all tags at once, not one by one #29798

Closed

Merge branch '2.4-develop' into 2.4-latest

2844fbf

engcom-Charlie self-assigned this Sep 21, 2020

engcom-Bravo requested changes Sep 28, 2020

View reviewed changes

ghost added Progress: needs update and removed Progress: testing in progress labels Sep 28, 2020

engcom-Bravo removed the Progress: needs update label Oct 5, 2020

ghost added the Progress: needs update label Oct 5, 2020

engcom-Bravo requested review from BarnyShergold and rogyar October 6, 2020 07:49

ghost added Progress: pending review and removed Progress: needs update labels Oct 6, 2020

rogyar reviewed Oct 6, 2020

View reviewed changes

rogyar approved these changes Oct 6, 2020

View reviewed changes

ghost added Progress: ready for testing and removed Progress: pending review labels Oct 6, 2020

ghost added Progress: testing in progress and removed Progress: ready for testing labels Oct 7, 2020

engcom-Bravo added the QA: Added to Regression Scope Scenario was analysed and added to Regression Testing Scope label Oct 7, 2020

ghost added Progress: extended testing Progress: accept and removed Progress: testing in progress Progress: extended testing labels Oct 7, 2020

magento-engcom-team pushed a commit that referenced this pull request Oct 7, 2020

ENGCOM-8237: Prevent flushing same tags several times #29885

5209417

magento-engcom-team merged commit 10c9331 into magento:2.4-develop Oct 7, 2020

ihor-sviziev mentioned this pull request Apr 19, 2021

Cache cleaner performance issue when saving multiple products #29275

Closed

5 tasks

Prevent flushing same tags several times #29885

Prevent flushing same tags several times #29885

Uh oh!

Conversation

siimm commented Sep 3, 2020 • edited by m2-assistant bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description (*)

Related Pull Requests

Fixed Issues (if relevant)

Manual testing scenarios (*)

Questions or comments

Contribution checklist (*)

Resolved issues:

Uh oh!

m2-assistant bot commented Sep 3, 2020

Uh oh!

sidolov commented Sep 3, 2020

Uh oh!

rogyar commented Sep 3, 2020

Uh oh!

siimm commented Sep 4, 2020

Uh oh!

lenaorobei commented Sep 4, 2020

Uh oh!

Uh oh!

Uh oh!

gabrieldagama commented Sep 7, 2020

Uh oh!

lenaorobei commented Sep 8, 2020

Uh oh!

ihor-sviziev commented Sep 17, 2020

Uh oh!

engcom-Bravo left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

siimm commented Oct 5, 2020

Uh oh!

engcom-Bravo commented Oct 5, 2020

Manual Testing scenario

Uh oh!

engcom-Bravo commented Oct 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rogyar Oct 6, 2020

Choose a reason for hiding this comment

Uh oh!

magento-engcom-team commented Oct 6, 2020

Uh oh!

m2-assistant bot commented Oct 7, 2020

Uh oh!

Uh oh!

siimm commented Sep 3, 2020 •

edited by m2-assistant bot

Loading

engcom-Bravo left a comment •

edited

Loading

engcom-Bravo commented Oct 6, 2020 •

edited

Loading