Investigate performance of next_version_mini_installer step on windows builder |
|||
Issue descriptionContext: https://chromium-review.googlesource.com/c/chromium/src/+/897235#message-e228a794625a58dc92a6736a730bb150d901d3ea After the land of above CL, next_version_mini_installer.exe can be critical path of windows build. https://chromium-build-stats.appspot.com/ninja_log/2018/02/02/win1359-c4/ninja_log.win1359-c4.chrome-bot.20180202-002216.5628.gz/trace.html thakis gave me some insight around the step. I'll investigate the step of performance and whether it is possible to incremental compression or not.
,
Feb 5 2018
In create_installer_archive.py script, below command for chrome.packed.7z takes 61 seconds out of 62 seconds. ['.\\..\\..\\third_party\\lzma_sdk\\Executable\\7za.exe', 'a', '-t7z', '-m0=BCJ2', '-m1=LZMA:d27:fb128', '-m2=LZMA:d22:fb128:mf=bt2', '-m3=LZMA:d22:fb128:mf=bt2', '-mb0:1', '-mb0s1:2', '-mb0s2:3', 'C:\\src\\chromium\\src\\out\\win7_chromium_rel_ng\\chrome.packed.7z', 'C:\\src\\chromium\\src\\out\\win7_chromium_rel_ng\\chrome.7z'] In alternate_version_generator, below commands took 71 seconds out of 75 seconds. "C:\src\chromium\src\out\win7_chromium_rel_ng\..\..\third_party\lzma_sdk\Executable\7za.exe" a -bd -t7z "C:\Users\tikuta\AppData\Local\Temp\mini_installer_test_temp171040_18132\CHROME.PACKED.7Z" "C:\Users\tikuta\AppData\Local\Temp\mini_installer_test_temp171040_18132\chrome.7z" -mx9
,
Feb 14 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/58d3ed627cc389ed7452321fc2e6378322279f8c commit 58d3ed627cc389ed7452321fc2e6378322279f8c Author: Takuto Ikuta <tikuta@chromium.org> Date: Wed Feb 14 01:57:02 2018 Disable chrome archive compression in non official build This CL makes chrome.7z and next_version_mini_installer.exe step fast on non-official builder. Main target is win7_chromium_rel_ng, which is currently the slowest tryserver in CQ. chrome.7z: 221,655 ms -> 1,715 ms next_version_mini_installer.exe: 113,951 ms -> 10,085 ms I took numbers from below traces. with this CL: https://chromium-build-stats.appspot.com/ninja_log/2018/02/05/win94-c4/ninja_log.win94-c4.chrome-bot.20180205-021816.8460.gz/trace.html from https://ci.chromium.org/buildbot/tryserver.chromium.win/win7_chromium_rel_ng/95878 without this CL: https://chromium-build-stats.appspot.com/ninja_log/2018/02/02/win1359-c4/ninja_log.win1359-c4.chrome-bot.20180202-002216.5628.gz/trace.html from https://ci.chromium.org/buildbot/tryserver.chromium.win/win7_chromium_rel_ng/94733 If we want to try CLs in is_official_build = true, we can use a builder having similar configs with official builder. https://ci.chromium.org/buildbot/tryserver.chromium.win/win_chrome_official/ Bug: 808910 Change-Id: I116b219ab756e16086d59a7d5c3c79872610faa7 Reviewed-on: https://chromium-review.googlesource.com/901402 Commit-Queue: Takuto Ikuta <tikuta@chromium.org> Reviewed-by: Gabriel Charette <gab@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Cr-Commit-Position: refs/heads/master@{#536601} [modify] https://crrev.com/58d3ed627cc389ed7452321fc2e6378322279f8c/chrome/installer/mini_installer/BUILD.gn [modify] https://crrev.com/58d3ed627cc389ed7452321fc2e6378322279f8c/chrome/installer/mini_installer/mini_installer.cc [modify] https://crrev.com/58d3ed627cc389ed7452321fc2e6378322279f8c/chrome/installer/mini_installer/mini_installer_constants.cc [modify] https://crrev.com/58d3ed627cc389ed7452321fc2e6378322279f8c/chrome/installer/mini_installer/mini_installer_constants.h [modify] https://crrev.com/58d3ed627cc389ed7452321fc2e6378322279f8c/chrome/tools/build/win/create_installer_archive.py
,
Feb 22 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e7220751085e18360ce44dc0ae3645c55f926f47 commit e7220751085e18360ce44dc0ae3645c55f926f47 Author: Takuto Ikuta <tikuta@chromium.org> Date: Thu Feb 22 03:01:31 2018 Support archive compression on component build This is partial revert of https://chromium-review.googlesource.com/c/chromium/src/+/901402/28..32 Bug: 808910 Change-Id: I66e90cf61d03df57912ec2507b142a5646ec6f06 Reviewed-on: https://chromium-review.googlesource.com/927870 Reviewed-by: Gabriel Charette <gab@chromium.org> Reviewed-by: Greg Thompson <grt@chromium.org> Commit-Queue: Takuto Ikuta <tikuta@chromium.org> Cr-Commit-Position: refs/heads/master@{#538326} [modify] https://crrev.com/e7220751085e18360ce44dc0ae3645c55f926f47/chrome/installer/mini_installer/BUILD.gn [modify] https://crrev.com/e7220751085e18360ce44dc0ae3645c55f926f47/chrome/installer/mini_installer/mini_installer.cc
,
Feb 24 2018
tikuta@, what's the gain in using the uncompressed mini_installer? Previously it was around 40MB, now it's more than 160MB which takes 4x more time to download. Now it's even larger than chrome-win32.zip (120MB). Generating the installer is a one time activity whereas downloading is performed lots of times.
,
Feb 25 2018
My CL is for improve compile step time on tryserver. Compression should be enabled on official build. Where do you download such large chrome-win32.zip from?
,
Feb 25 2018
The 120 MB zip was always used by bisect_builds.py script. I've modified it to use the 40 MB mini installer exe to reduce the download size. Bit that's just my case. If you have download statistics for the snapshots server [1], you can see how many times the world uses that exe. [1]: https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Win/510820/
,
Feb 25 2018
More links just in case: 1. bisect_builds.py https://www.chromium.org/developers/bisect-builds-py 2. r536601 increased the mini installer to 185MB (x64) or 170MB (x86): https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Win_x64/536601/ https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Win/539018/
,
Feb 25 2018
FWIW, to help the build bot you can speed up 7zip an order of magnitude (or even two) by switching to a newer 7zip. * use LZMA2 * use fast mode (a0) * reduce the dictionary size to 64kB * increase threads to 16 7z.exe a -m1=LZMA2:a0:d64k:mt16 chrome.7z No other switches are necessary. Note, it's 7z.exe now, not 7za.exe. It takes 2 seconds (yes, two!) here to compress the Chrome app directory. The archive size will be ~60MB, which is larger than the historical 40MB-50MB, but still 3 times smaller than uncompressed.
,
Feb 27 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/90a97ac8e82131984240d6084f6c5f9b24b31288 commit 90a97ac8e82131984240d6084f6c5f9b24b31288 Author: Takuto Ikuta <tikuta@google.com> Date: Tue Feb 27 00:34:30 2018 Enable archive compression on waterfall win builder This CL enables archive compression on waterfall builder to revive the smaller size of installer binary which is downloaded by bisect script many times. Size reduction will be 185MB -> 40MB on x64 build. https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Win_x64/536587/ https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Win_x64/536601/ Bug: 808910 Change-Id: I34bbb2e3c12a380ba222a8749ee720d8b896e4c8 Reviewed-on: https://chromium-review.googlesource.com/936344 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Takuto Ikuta <tikuta@google.com> Cr-Commit-Position: refs/heads/master@{#539346} [modify] https://crrev.com/90a97ac8e82131984240d6084f6c5f9b24b31288/tools/mb/mb_config.pyl
,
Feb 28 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e25b36794b4528cf0d80f312b4f03b0bcaaa8df9 commit e25b36794b4528cf0d80f312b4f03b0bcaaa8df9 Author: Takuto Ikuta <tikuta@google.com> Date: Wed Feb 28 01:10:09 2018 Fix gn_args enable_archive_compression -> skip_archive_compression I used wrong config flag name in https://chromium-review.googlesource.com/936344 skip_archive_compression is the flag I need to set. https://chromium-review.googlesource.com/901402 Bug: 808910 Change-Id: I01118ee9081dddfec6d802c860e41434682afcb2 Reviewed-on: https://chromium-review.googlesource.com/939181 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Takuto Ikuta <tikuta@google.com> Cr-Commit-Position: refs/heads/master@{#539603} [modify] https://crrev.com/e25b36794b4528cf0d80f312b4f03b0bcaaa8df9/tools/mb/mb_config.pyl
,
Feb 28 2018
Size of mini_installer become smaller again. https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Win_x64/539687/
,
Feb 28 2018
tikuta: thoughts on woxxom's suggestion to update the 7zip version we use? That sounds like a good potential easy win...
,
Mar 1 2018
#9, #13 Let me file separate issue for this. https://bugs.chromium.org/p/chromium/issues/detail?id=817699 This is not in my prioritized list now.
,
Mar 28 2018
|
|||
►
Sign in to add a comment |
|||
Comment 1 by tikuta@chromium.org
, Feb 5 201855.8 KB
55.8 KB View Download