Speedup static content deploy for specific theme #30090

ihor-sviziev · 2020-09-17T13:27:50Z

Description (*)

This PR adds improvement for ability to deploy the only specific theme(s) for multiple locales, so it should improve build phase of deployment for the those who use single theme with different locales.

Basically what I did - i just removed packages from all themes from which inherited requested theme, so in case of Luma - we have only 1 parent theme, but for people who has bigger inheritance - different will be bigger.

How I tested:

Before:

php bin/magento setup:upgrade
php bin/magento setup:static-content:deploy -f -t Magento/luma -- en_AU en_CA en_IE en_NZ en_GB en_US

~114 seconds ❌

After:

php bin/magento setup:upgrade
php bin/magento setup:static-content:deploy -f -t Magento/luma --no-parent -- en_AU en_CA en_IE en_NZ en_GB en_US

~60 seconds ✔

Related Pull Requests

Fixed Issues (if relevant)

N/A

Manual testing scenarios (*)

...
...

Questions or comments

Contribution checklist (*)

Pull request has a meaningful description of its purpose
All commits are accompanied by meaningful commit messages
All new or changed code is covered with unit/integration tests (if applicable)
All automated tests passed successfully (all builds are green)

Resolved issues:

resolves [Issue] Speedup static content deploy for specific theme #30184: Speedup static content deploy for specific theme

m2-assistant · 2020-09-17T13:27:53Z

Hi @ihor-sviziev. Thank you for your contribution
Here is some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

@magento give me test instance - deploy test instance based on PR changes
@magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

@magento run all tests - run or re-run all required tests against the PR changes
@magento run <test-build(s)> - run or re-run specific test build(s)
For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names. Allowed build names are:

Database Compare
Functional Tests CE
Functional Tests EE,
Functional Tests B2B
Integration Tests
Magento Health Index
Sample Data Tests CE
Sample Data Tests EE
Sample Data Tests B2B
Static Tests
Unit Tests
WebAPI Tests

You can find more information about the builds here

ℹ️ Please run only needed test builds instead of all when developing. Please run all test builds before sending your PR for review.

For more details, please, review the Magento Contributor Guide documentation.

⚠️ According to the Magento Contribution requirements, all Pull Requests must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of Pull Requests happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

ihor-sviziev · 2020-09-17T13:30:42Z

app/code/Magento/Deploy/Strategy/QuickDeploy.php

+     */
+    private function canDeployTheme(string $theme, array $includedThemesMap, array $excludedEntitiesMap): bool
+    {
+        $includesAllThemes = \array_key_exists('all', $includedThemesMap);


Using here \array_key_exists instead of isset because of optimization php/php-src#3360 that was done in php 7.4, and definitely not in_array because of performance issues.

ihor-sviziev · 2020-09-17T13:32:35Z

@magento run all tests

ihor-sviziev · 2020-09-17T14:34:27Z

@magento run all tests

hostep · 2020-09-17T16:41:57Z

Nice! So this will revert the SCD behavior back to how it was in Magento 2.1 from memory.

It might cause performance decreases for deploying multiple themes which inherit from the same parent theme (since it might not generate the parent theme files and can't re-use them), but I'm not sure. If that's the case, you can probably first deploy the parent theme followed by the child themes, that should result in the same performance probably.

I'm assuming you tested that the resulting generated files are 100% the same as before this change?

ihor-sviziev · 2020-09-17T16:55:51Z

@hostep i didn’t find any difference in result and in performance between generating multiple themes, or single theme

hostep · 2020-09-17T18:34:46Z

Cool!

Regarding performance, I'm curious if there will be a difference if we have a theme structure like:

parent theme
 - child theme 1
 - child theme 2
 - child theme 3

Also: should the same change be added to the StandardDeploy and CompactDeploy classes?

ihor-sviziev · 2020-09-18T06:28:58Z

@hostep actually I never used them, might check that if needed, but not sure if it will work for them.

mrtuvn · 2020-09-19T02:57:22Z

Maybe not related in scope of this but i'm just curious about the use case magento site setup multiple themes (around 15 themes included magento theme base) and various some locales (2-3 locales). When owner run deploy all theme for 3 locales! No exclude theme specific

ihor-sviziev · 2020-09-19T06:25:34Z

@mrtuvn in this case it will work as it was before ✅

BarnyShergold · 2020-09-21T15:27:14Z

How will this affect the build phase for Magento Cloud where we can control what themes are built and what locales? Or will it not have an effect?

ihor-sviziev · 2020-09-21T15:44:08Z

@BarnyShergold unfortunately I don’t have access to ee and magento cloud so can’t check it, but I think it should work fine as it’s basically runs magento static content deploy command with specified list of themes and locales, the only change will be - the only requested themes will be compiled.

Could you help to check if it works fine on Magento Cloud?

ihor-sviziev · 2020-09-24T10:31:24Z

@magento create issue

NadiyaS · 2020-09-25T22:17:53Z

Hi,
There are some results after testing this implementation on Cloud integration environment.
SCD_MATRIX configuration was used to test SCD with separate CLI command for themes.
Generation of luma theme was faster from ~ 17% up to 45% (depends on configurations in SCD_MATRIX).
E.g.

  build:
    SCD_MATRIX:
      Magento/blank:
        language:
          - fr_CA
          - fr_FR
          - nl_NL
          - de_DE
          - en_CA
          - en_US
      Magento/luma:
        language:
          - en_US
          - en_AU
          - en_IE 
          - en_NZ 
          - en_GB
          - en_CA

For this configuration ece-tools runs 3 CLI commands:

php ./bin/magento setup:static-content:deploy --ansi --no-interaction -f --jobs 4 --no-html-minify --exclude-theme Magento/luma --exclude-theme Magento/blank en_US en_AU en_IE en_NZ en_GB en_CA
php ./bin/magento setup:static-content:deploy --ansi --no-interaction -f --jobs 4 --no-html-minify --theme Magento/blank fr_CA fr_FR nl_NL de_DE en_CA en_US
php ./bin/magento setup:static-content:deploy --ansi --no-interaction -f --jobs 4 --no-html-minify --theme Magento/luma en_US en_AU en_IE en_NZ en_GB en_CA
Time for first 2 commands took ~ 33 sec and 22 sec accordingly (as for new as for old implementation)
Generation of luma theme took 40 sec before changes and 25 sec after
Generation of all static content for all themes was on ~ 17% faster.

ihor-sviziev · 2020-10-05T07:32:47Z

Open questions:

Backward compatibility

Currently someone could depends on parent themes, so for these people this change.

Possible solutions:

add new optional param, for instance --no-parent to bin/magento setup:static-content:deploy command
merge this change to 2.5
...any other options?

Different behavior between the `quick`, `compact` and `standard` strategies

Currently my fix is improving only quick strategy, but BTH I never used any other strategies and don't really know if there any reason to use them.

From my perspective - it's better to support only one strategy and have this fix already merged.

YPyltiai · 2020-10-05T15:06:50Z

@ihor-sviziev , as discussed on Slack - option should be good to have to get this change into earlier versions. We can then adjust ECE-Tools to handle this flag and make it configurable through current configs. If this is confirmed working well - in 2.5 what flag does could be adjusted to become a default behavior.

hostep · 2020-10-05T16:04:39Z

I agree with the new --no-parent flag.

Regarding the different strategies, I would strongly recommend to try to apply the same change to the standard strategy, I think it's going to work (even though very few people use it probably). At the very least, we should make an attempt at trying this, if it fails and it causes too much headaches to get it sorted out, we can chose to not apply it.

The compact strategy is used more often (because it runs many times faster then standard or quick, but the downside is that it adds extra javascript overhead on the frontend of the shop and that certain 3rd party modules will break when using it).
But, I'm not sure if the same change can be applied due to how it works, it might need the parent theme to be deployed as well, but I'm not sure...

Also: if the change doesn't apply to one or two strategies, the description of this new --no-parent flag should highlight that it only affects certain strategies.

…cific-theme

gabrieldagama · 2020-11-09T10:38:11Z

@magento run all tests

ihor-sviziev · 2020-11-11T08:00:22Z

@magento run Functional Tests B2B

ihor-sviziev · 2020-11-13T07:52:51Z

@magento run Functional Tests B2B

@gabrieldagama any reasons why this PR isn't getting merged?

gabrieldagama · 2020-11-13T15:22:00Z

@ihor-sviziev we were having some failed tests that I had suspected could be related to it. I'm trying to deliver it again, hopefully, we will merge it soon.

m2-assistant · 2020-11-15T13:12:36Z

Hi @ihor-sviziev, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.

mrtuvn · 2021-06-22T12:15:57Z

Hi guys just curiously how fast we can get with this changes in specific case we have multiple themes + multiple locales in magento cloud? Anyone have benchmark this case
SCD_THREADS 4 SCD_STRATEGY: standard SCD_COMPRESSION_LEVEL: 4 SCD_ON_DEMAND: false
Frontend total themes 20 themes individual based blank. Each theme target for one locale. Backend theme will have deploy for all locales used!
CC: @ihor-sviziev @gabrieldagama @hostep @NadiyaS

ihor-sviziev · 2021-06-22T14:00:17Z

@mrtuvn sorry, I don't have

hostep · 2021-06-22T20:26:33Z

@mrtuvn: I can't speak about cloud, but just about Magento OS 2.4.2-p1 on premise.

Last week I did some optimisations to a certain project's deploy flow where we went from 183.5 seconds to 36.5 seconds (only for the SCD commands that is). This was using 2 different frontend themes that both had two levels of parent themes, 4 different locale's got deployed for the frontend and the Magento backend theme only got deployed in one locale.

We accomplished this by:

using 4 threads instead of 1 (--jobs flag)
using the new --no-parent flag introduced here (again, very big thank you to @ihor-sviziev for this!)
also used the --no-js-bundle flag, because we don't use js bundles on this particular project (this does shave of a few seconds believe it or not)
removing the web/tailwind directory in a custom frontend theme, between compiling tailwind css and running the Magento SCD command, otherwise the SCD command processes a bunch of files it shouldn't (this particular project uses Hyvä frontend technology, which uses tailwind) - this is no longer needed on more recent Hyvä versions

Not sure if these are really helpful numbers, because of the particularities of this project, but it's something 🙂

mrtuvn · 2021-06-23T01:47:34Z

thanks you very much for useful infor
interesting use-cases for no-parent flag
https://devdocs.magento.com/cloud/env/variables-build.html#scd_no_parent
also flag scd_use_baler too. This one supprised me when cloud allow we setup baler. Maybe this one already outdates

mrtuvn · 2021-06-24T02:56:05Z

Btw seem this fix not delivered to 2.3.7! Can it work for 2.3

ihor-sviziev · 2021-06-24T07:02:25Z

@mrtuvn, we just applied changes from this PR as a patch and using it in 2.3 release line. 👍

mrtuvn · 2022-08-10T00:10:47Z

@magento convert to patch for 2.3.7

m2-github-services · 2022-08-10T00:11:40Z

@mrtuvn changes has been successfully converted for 2.3.7 version and pushed to the converted-magento-magento2-30090-2.3.7 branch.

You may submit a pull request using the link: 2.3.7-release...magento:converted-magento-magento2-30090-2.3.7

HenKun · 2022-09-21T09:08:33Z

@magento convert to patch for 2.4.1

m2-github-services · 2022-09-21T09:08:38Z

@HenKun an error occurred during the patch generation.
Failure can be caused by the incompatible with base branch changes.
To replicate the issue try to apply patch locally using git am command.

ghost added the Progress: pending review label Sep 17, 2020

magento-engcom-team added Component: Deploy Release Line: 2.4 labels Sep 17, 2020

ihor-sviziev commented Sep 17, 2020

View reviewed changes

ghost assigned ihor-sviziev Sep 17, 2020

ihor-sviziev removed their assignment Sep 17, 2020

ihor-sviziev force-pushed the improve-static-content-deploy-for-specific-theme branch from 625607c to afbb8be Compare September 17, 2020 14:34

sidolov added Priority: P3 May be fixed according to the position in the backlog. Risk: medium Severity: S3 Affects non-critical data or functionality and does not force users to employ a workaround. labels Sep 17, 2020

ihor-sviziev requested a review from YPyltiai September 21, 2020 06:54

m2-assistant bot mentioned this pull request Sep 24, 2020

[Issue] Speedup static content deploy for specific theme #30184

Closed

YPyltiai requested a review from NadiyaS September 24, 2020 16:43

YPyltiai mentioned this pull request Oct 5, 2020

Adjust SCD Process & Variables to Work with New SCD Improvements magento/ece-tools#785

Closed

engcom-Bravo added the Auto-Tests: Covered All changes in Pull Request is covered by auto-tests label Nov 3, 2020

ghost added Progress: testing in progress Progress: accept and removed Progress: testing in progress Progress: ready for testing labels Nov 3, 2020

Merge branch '2.4-develop' into improve-static-content-deploy-for-spe…

553dfdb

…cific-theme

magento-engcom-team pushed a commit that referenced this pull request Nov 15, 2020

ENGCOM-8419: Speedup static content deploy for specific theme #30090

12feeff

magento-engcom-team merged commit f638d19 into magento:2.4-develop Nov 15, 2020

ihor-sviziev deleted the improve-static-content-deploy-for-specific-theme branch November 29, 2020 00:34

ihor-sviziev mentioned this pull request Jan 6, 2021

Why does magento try to compile inactive themes? #29628

Closed

5 tasks

hostep mentioned this pull request Jan 25, 2021

[2.4.2] Document the new --no-parent flag in the static content deploy command magento/devdocs#8561

Closed

hostep mentioned this pull request Feb 11, 2021

setup:static-content:deploy fails with errors which makes no sense #32075

Closed

5 tasks

chandru-199 mentioned this pull request Mar 3, 2021

Added new option to static-content:deploy command magento/devdocs#8765

Merged

sprankhub mentioned this pull request Apr 27, 2022

Magento 2: Add Support For --no-parent deployphp/deployer#3117

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Speedup static content deploy for specific theme #30090

Speedup static content deploy for specific theme #30090

ihor-sviziev commented Sep 17, 2020 •

edited

Loading

m2-assistant bot commented Sep 17, 2020

ihor-sviziev Sep 17, 2020 •

edited

Loading

ihor-sviziev commented Sep 17, 2020

ihor-sviziev commented Sep 17, 2020

hostep commented Sep 17, 2020

ihor-sviziev commented Sep 17, 2020

hostep commented Sep 17, 2020

ihor-sviziev commented Sep 18, 2020

mrtuvn commented Sep 19, 2020 •

edited

Loading

ihor-sviziev commented Sep 19, 2020

BarnyShergold commented Sep 21, 2020

ihor-sviziev commented Sep 21, 2020 •

edited

Loading

ihor-sviziev commented Sep 24, 2020

NadiyaS commented Sep 25, 2020

ihor-sviziev commented Oct 5, 2020

YPyltiai commented Oct 5, 2020

hostep commented Oct 5, 2020

gabrieldagama commented Nov 9, 2020

ihor-sviziev commented Nov 11, 2020

ihor-sviziev commented Nov 13, 2020

gabrieldagama commented Nov 13, 2020

m2-assistant bot commented Nov 15, 2020

mrtuvn commented Jun 22, 2021 •

edited

Loading

ihor-sviziev commented Jun 22, 2021

hostep commented Jun 22, 2021 •

edited

Loading

mrtuvn commented Jun 23, 2021 •

edited

Loading

mrtuvn commented Jun 24, 2021

ihor-sviziev commented Jun 24, 2021

mrtuvn commented Aug 10, 2022

m2-github-services commented Aug 10, 2022

HenKun commented Sep 21, 2022

m2-github-services commented Sep 21, 2022

Speedup static content deploy for specific theme #30090

Speedup static content deploy for specific theme #30090

Conversation

ihor-sviziev commented Sep 17, 2020 • edited Loading

Description (*)

Related Pull Requests

Fixed Issues (if relevant)

Manual testing scenarios (*)

Questions or comments

Contribution checklist (*)

Resolved issues:

m2-assistant bot commented Sep 17, 2020

ihor-sviziev Sep 17, 2020 • edited Loading

Choose a reason for hiding this comment

ihor-sviziev commented Sep 17, 2020

ihor-sviziev commented Sep 17, 2020

hostep commented Sep 17, 2020

ihor-sviziev commented Sep 17, 2020

hostep commented Sep 17, 2020

ihor-sviziev commented Sep 18, 2020

mrtuvn commented Sep 19, 2020 • edited Loading

ihor-sviziev commented Sep 19, 2020

BarnyShergold commented Sep 21, 2020

ihor-sviziev commented Sep 21, 2020 • edited Loading

ihor-sviziev commented Sep 24, 2020

NadiyaS commented Sep 25, 2020

ihor-sviziev commented Oct 5, 2020

Backward compatibility

Different behavior between the quick, compact and standard strategies

YPyltiai commented Oct 5, 2020

hostep commented Oct 5, 2020

gabrieldagama commented Nov 9, 2020

ihor-sviziev commented Nov 11, 2020

ihor-sviziev commented Nov 13, 2020

gabrieldagama commented Nov 13, 2020

m2-assistant bot commented Nov 15, 2020

mrtuvn commented Jun 22, 2021 • edited Loading

ihor-sviziev commented Jun 22, 2021

hostep commented Jun 22, 2021 • edited Loading

mrtuvn commented Jun 23, 2021 • edited Loading

mrtuvn commented Jun 24, 2021

ihor-sviziev commented Jun 24, 2021

mrtuvn commented Aug 10, 2022

m2-github-services commented Aug 10, 2022

HenKun commented Sep 21, 2022

m2-github-services commented Sep 21, 2022

ihor-sviziev commented Sep 17, 2020 •

edited

Loading

ihor-sviziev Sep 17, 2020 •

edited

Loading

mrtuvn commented Sep 19, 2020 •

edited

Loading

ihor-sviziev commented Sep 21, 2020 •

edited

Loading

Different behavior between the `quick`, `compact` and `standard` strategies

mrtuvn commented Jun 22, 2021 •

edited

Loading

hostep commented Jun 22, 2021 •

edited

Loading

mrtuvn commented Jun 23, 2021 •

edited

Loading