New issue
Advanced search Search tips

Issue 808910 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Feb 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 3
Type: Bug

Blocking:
issue 804251



Sign in to add a comment

Investigate performance of next_version_mini_installer step on windows builder

Project Member Reported by tikuta@chromium.org, Feb 5 2018

Issue description

Context:
https://chromium-review.googlesource.com/c/chromium/src/+/897235#message-e228a794625a58dc92a6736a730bb150d901d3ea

After the land of above CL, next_version_mini_installer.exe can be critical path of windows build.
https://chromium-build-stats.appspot.com/ninja_log/2018/02/02/win1359-c4/ninja_log.win1359-c4.chrome-bot.20180202-002216.5628.gz/trace.html

thakis gave me some insight around the step.
I'll investigate the step of performance and whether it is possible to incremental compression or not.
 
I built next_version_mini_installer with win7_chromium_rel_ng config in local machine.

Generating chrome.7z, setup.ex_, gen/chrome/installer/mini_installer/mini_installer/packed_files.rc, chrome.packed.7z took 62 seconds and next_version_mini_installer.exe took 75 seconds.

I will break down each step more. And considering to use multi-thread compression for 7za.
TMCGzb7VXJ2.png
55.8 KB View Download
In create_installer_archive.py script, below command for chrome.packed.7z takes 61 seconds out of 62 seconds.
['.\\..\\..\\third_party\\lzma_sdk\\Executable\\7za.exe', 'a', '-t7z', '-m0=BCJ2', '-m1=LZMA:d27:fb128', '-m2=LZMA:d22:fb128:mf=bt2', '-m3=LZMA:d22:fb128:mf=bt2', '-mb0:1', '-mb0s1:2', '-mb0s2:3', 'C:\\src\\chromium\\src\\out\\win7_chromium_rel_ng\\chrome.packed.7z', 'C:\\src\\chromium\\src\\out\\win7_chromium_rel_ng\\chrome.7z']


In alternate_version_generator, below commands took 71 seconds out of 75 seconds.
"C:\src\chromium\src\out\win7_chromium_rel_ng\..\..\third_party\lzma_sdk\Executable\7za.exe" a -bd -t7z "C:\Users\tikuta\AppData\Local\Temp\mini_installer_test_temp171040_18132\CHROME.PACKED.7Z" "C:\Users\tikuta\AppData\Local\Temp\mini_installer_test_temp171040_18132\chrome.7z" -mx9
Project Member

Comment 3 by bugdroid1@chromium.org, Feb 14 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/58d3ed627cc389ed7452321fc2e6378322279f8c

commit 58d3ed627cc389ed7452321fc2e6378322279f8c
Author: Takuto Ikuta <tikuta@chromium.org>
Date: Wed Feb 14 01:57:02 2018

Disable chrome archive compression in non official build

This CL makes chrome.7z and next_version_mini_installer.exe step fast on non-official builder.
Main target is win7_chromium_rel_ng, which is currently the slowest tryserver in CQ.

chrome.7z: 221,655 ms -> 1,715 ms
next_version_mini_installer.exe: 113,951 ms -> 10,085 ms

I took numbers from below traces.
with this CL: https://chromium-build-stats.appspot.com/ninja_log/2018/02/05/win94-c4/ninja_log.win94-c4.chrome-bot.20180205-021816.8460.gz/trace.html
from https://ci.chromium.org/buildbot/tryserver.chromium.win/win7_chromium_rel_ng/95878

without this CL: https://chromium-build-stats.appspot.com/ninja_log/2018/02/02/win1359-c4/ninja_log.win1359-c4.chrome-bot.20180202-002216.5628.gz/trace.html
from https://ci.chromium.org/buildbot/tryserver.chromium.win/win7_chromium_rel_ng/94733

If we want to try CLs in is_official_build = true, we can use a builder having similar configs with official builder.
https://ci.chromium.org/buildbot/tryserver.chromium.win/win_chrome_official/

Bug:  808910 
Change-Id: I116b219ab756e16086d59a7d5c3c79872610faa7
Reviewed-on: https://chromium-review.googlesource.com/901402
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Gabriel Charette <gab@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#536601}
[modify] https://crrev.com/58d3ed627cc389ed7452321fc2e6378322279f8c/chrome/installer/mini_installer/BUILD.gn
[modify] https://crrev.com/58d3ed627cc389ed7452321fc2e6378322279f8c/chrome/installer/mini_installer/mini_installer.cc
[modify] https://crrev.com/58d3ed627cc389ed7452321fc2e6378322279f8c/chrome/installer/mini_installer/mini_installer_constants.cc
[modify] https://crrev.com/58d3ed627cc389ed7452321fc2e6378322279f8c/chrome/installer/mini_installer/mini_installer_constants.h
[modify] https://crrev.com/58d3ed627cc389ed7452321fc2e6378322279f8c/chrome/tools/build/win/create_installer_archive.py

Project Member

Comment 4 by bugdroid1@chromium.org, Feb 22 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e7220751085e18360ce44dc0ae3645c55f926f47

commit e7220751085e18360ce44dc0ae3645c55f926f47
Author: Takuto Ikuta <tikuta@chromium.org>
Date: Thu Feb 22 03:01:31 2018

Support archive compression on component build

This is partial revert of
https://chromium-review.googlesource.com/c/chromium/src/+/901402/28..32

Bug:  808910 
Change-Id: I66e90cf61d03df57912ec2507b142a5646ec6f06
Reviewed-on: https://chromium-review.googlesource.com/927870
Reviewed-by: Gabriel Charette <gab@chromium.org>
Reviewed-by: Greg Thompson <grt@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Cr-Commit-Position: refs/heads/master@{#538326}
[modify] https://crrev.com/e7220751085e18360ce44dc0ae3645c55f926f47/chrome/installer/mini_installer/BUILD.gn
[modify] https://crrev.com/e7220751085e18360ce44dc0ae3645c55f926f47/chrome/installer/mini_installer/mini_installer.cc

Comment 5 by woxxom@gmail.com, Feb 24 2018

tikuta@, what's the gain in using the uncompressed mini_installer? Previously it was around 40MB, now it's more than 160MB which takes 4x more time to download. Now it's even larger than chrome-win32.zip (120MB). Generating the installer is a one time activity whereas downloading is performed lots of times.

Comment 6 by tikuta@chromium.org, Feb 25 2018

My CL is for improve compile step time on tryserver.
Compression should be enabled on official build.
Where do you download such large chrome-win32.zip from?

Comment 7 by woxxom@gmail.com, Feb 25 2018

The 120 MB zip was always used by bisect_builds.py script. I've modified it to use the 40 MB mini installer exe to reduce the download size. Bit that's just my case. If you have download statistics for the snapshots server [1], you can see how many times the world uses that exe. 

  [1]: https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Win/510820/

Comment 9 by woxxom@gmail.com, Feb 25 2018

FWIW, to help the build bot you can speed up 7zip an order of magnitude (or even two) by switching to a newer 7zip.
* use LZMA2 
* use fast mode (a0)
* reduce the dictionary size to 64kB
* increase threads to 16

7z.exe a -m1=LZMA2:a0:d64k:mt16 chrome.7z

No other switches are necessary. Note, it's 7z.exe now, not 7za.exe.
It takes 2 seconds (yes, two!) here to compress the Chrome app directory.
The archive size will be ~60MB, which is larger than the historical 40MB-50MB, but still 3 times smaller than uncompressed.
Project Member

Comment 10 by bugdroid1@chromium.org, Feb 27 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/90a97ac8e82131984240d6084f6c5f9b24b31288

commit 90a97ac8e82131984240d6084f6c5f9b24b31288
Author: Takuto Ikuta <tikuta@google.com>
Date: Tue Feb 27 00:34:30 2018

Enable archive compression on waterfall win builder

This CL enables archive compression on waterfall builder to revive the smaller size of installer binary which is downloaded by bisect script many times.

Size reduction will be 185MB -> 40MB on x64 build.
https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Win_x64/536587/
https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Win_x64/536601/

Bug:  808910 
Change-Id: I34bbb2e3c12a380ba222a8749ee720d8b896e4c8
Reviewed-on: https://chromium-review.googlesource.com/936344
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@google.com>
Cr-Commit-Position: refs/heads/master@{#539346}
[modify] https://crrev.com/90a97ac8e82131984240d6084f6c5f9b24b31288/tools/mb/mb_config.pyl

Project Member

Comment 11 by bugdroid1@chromium.org, Feb 28 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e25b36794b4528cf0d80f312b4f03b0bcaaa8df9

commit e25b36794b4528cf0d80f312b4f03b0bcaaa8df9
Author: Takuto Ikuta <tikuta@google.com>
Date: Wed Feb 28 01:10:09 2018

Fix gn_args enable_archive_compression -> skip_archive_compression

I used wrong config flag name in https://chromium-review.googlesource.com/936344
skip_archive_compression is the flag I need to set.
https://chromium-review.googlesource.com/901402

Bug:  808910 
Change-Id: I01118ee9081dddfec6d802c860e41434682afcb2
Reviewed-on: https://chromium-review.googlesource.com/939181
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@google.com>
Cr-Commit-Position: refs/heads/master@{#539603}
[modify] https://crrev.com/e25b36794b4528cf0d80f312b4f03b0bcaaa8df9/tools/mb/mb_config.pyl

Status: Fixed (was: Untriaged)
Size of mini_installer become smaller again.
https://commondatastorage.googleapis.com/chromium-browser-snapshots/index.html?prefix=Win_x64/539687/
tikuta: thoughts on woxxom's suggestion to update the 7zip version we use? That sounds like a good potential easy win...
#9, #13 
Let me file separate issue for this.
https://bugs.chromium.org/p/chromium/issues/detail?id=817699
This is not in my prioritized list now.
Blocking: 804251

Sign in to add a comment