Many cron processes can wait locks and consume resources

[ilnytskyi]

Preconditions (*)

magento 2.3.x

1. Cron observer uses locks (db, cache, file or zookeper)
2. The job is locked after cron:run is sent to shell
   see this method
   \Magento\Cron\Observer\ProcessCronQueueObserver::execute

The problem, I suppose, that magetno uses the same method to run all groups or one group
in \Magento\Cron\Observer\ProcessCronQueueObserver::execute
it creates commands in shell

            if ($this->_request->getParam(self::STANDALONE_PROCESS_STARTED) !== '1'
                && $this->getCronGroupConfigurationValue($groupId, 'use_separate_process') == 1
            ) {
                $this->_shell->execute(
                    $phpPath . ' %s cron:run --group=' . $groupId . ' --' . Cli::INPUT_KEY_BOOTSTRAP . '='
                    . self::STANDALONE_PROCESS_STARTED . '=1',
                    [
                        BP . '/bin/magento'
                    ]
                );
                continue;
            }

and only then, it runs the command like this
php bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1
it goes to this part

            $this->lockGroup(
                $groupId,
                function ($groupId) use ($currentTime, $jobsRoot) {
                    $this->cleanupJobs($groupId, $currentTime);
                    $this->generateSchedules($groupId);
                    $this->processPendingJobs($groupId, $jobsRoot, $currentTime);
                }
            );

Technically possible to remove all locks and run many bin/magento cron:run & and see dozens of created processes in shell
e.g. indexer php bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1

Steps to reproduce (*)

1. Run bin/magento cron:run & a few times in background
2. See that it creates more than 1 process in shell
   eg php bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1

Expected result (*)

1. If in shell one cron command is running the second should not be created
2. only one unique shell command should be processed
3. Lock for process or group should be checked before the command goes to shell

Actual result (*)

1. Magento can create many processes at time
2. May simultaneously running processes check locks wait and consumes server resources
3. Lock for process or group is checked after the command is in shell
   

GIF:

WORKAROUND:
Install each cron group separetely with flock instead adding default cron:run in crontab.

[m2-assistant]

Hi @ilnytskyi. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

@ilnytskyi do you confirm that you were able to reproduce the issue on vanilla Magento instance following steps to reproduce?

• yes
• no

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[m2-assistant]

Hi @engcom-Charlie. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• 4. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• 5. Add label Issue: Confirmed once verification is complete.

• 6. Make sure that automatic system confirms that report has been added to the backlog.

[engcom-Charlie]

Hello @ilnytskyi
Thank you for contribution and collaboration!

We are not able to reproduce this issue on the latest 2.4-develop branch by provided steps.

Result:

• cron:run group shows same behavior like as cron:run
• in shell one cron command is running the second should not be created
• only one unique shell command should be processed
• Lock for process or group is checked before the command goes to shell
  

Could you verify my steps?

[ilnytskyi]

@engcom-Charlie
Yes sure
just run a few commands in background
php bin/magento cron:run & php bin/magento cron:run & php bin/magento cron:run & php bin/magento cron:run & php bin/magento cron:run & php bin/magento cron:run &

Lock for process or group is not checked before the command goes to shellt

You can see it on gif. Although on local fresh installation it goes fast this can be a problem on live server with high load.
And flock for bin/magento cron:run will not help since cron:run creates another processes that can wait for locks

mb smth like this for cron command can be used

'/usr/bin/flock -w 0 ' . BP . '/var/' . self::LOCK_PREFIX . $groupId . '.lock ' . $phpPath . ' %s cron:run --group=' . $groupId . ' --' . Cli::INPUT_KEY_BOOTSTRAP . '='

or this

            if ($this->lockManager->isLocked(self::LOCK_PREFIX . $groupId)) {
                continue;
            }

However I noticed some command still try to execute simultaneously in second solution, and using it in the loop can froze other jobs. Probably smth like first solution should be used on the shell level

[engcom-Charlie]

@ilnytskyi
Thank you for contribution and collaboration!
i just reproduced with your updates.
Now i can confirm this issue.