+Infra>Client>Chrome 

Clicky: https://bugs.chromium.org/p/angleproject/issues/detail?id=2793

This is a problem on linux too, the hooks are just faster there, but they still do everything on each build: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Deterministic%20Linux/17347 (still > 6 min)

Probably needs something similar to https://chromium-review.googlesource.com/c/chromium/tools/build/+/1203612 ; I'll give it a shot.

https://chromium-review.googlesource.com/c/chromium/tools/build/+/1296834

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/e9744f0663d2d04173336af56d9197ad501eb5e6

commit e9744f0663d2d04173336af56d9197ad501eb5e6
Author: Nico Weber <thakis@chromium.org>
Date: Thu Oct 25 10:57:51 2018

Set up builder and msvc cache for deterministic bots.

Currently, the deterministic bots download everything downloaded by hooks
on each build with takes 6 min on linux and 10 on windows. Set up a named
cache so that most stuff downloaded by hooks is cached. On Windows, msvc
gets downloaded into depot_tools by default (instead of into the named cache)
which is cleaned on each build, so also set up an env var that makes it so
that msvc gets cached.

Bug:  897854 
Change-Id: Ic4f4e2ecc5e8125cb0a68b672e4994ecd0f8fb27
Reviewed-on: https://chromium-review.googlesource.com/c/1296834
Auto-Submit: Nico Weber <thakis@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>

[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic.json
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic__dbg_.json
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux_fail.json
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic.json
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux.json
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android__dbg_.json
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg__fail.json
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic_fail.json
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg_.json
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android.json
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic__dbg__fail.json
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android__dbg__fail.json
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic_fail.json
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/README.recipes.md
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic.json
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android_fail.json
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic.json
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic_fail.json
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic_fail.json
[modify] https://crrev.com/e9744f0663d2d04173336af56d9197ad501eb5e6/scripts/slave/recipes/swarming/deterministic_build.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/4a6e603768d21161b37c19dfd7794848e06d475c

commit 4a6e603768d21161b37c19dfd7794848e06d475c
Author: Nico Weber <thakis@chromium.org>
Date: Thu Oct 25 17:49:08 2018

Make deterministic android recipe look more like the regular recipe

The Deterministic Android build started failing in weird ways after
I landed my build cache CL from yesterday:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Deterministic%20Android/2960

The main waterfall android bots seem to no longer use chromium_android directly
(no "cleanup index.lock" step there), so try using the regular
chromium recipe on the Deterministic Android build to see if that helps.

TBR=tandrii
Bug:  897854 
Change-Id: I76b5a95595fc74aa676369b33b2c08cdc2e0096c
Reviewed-on: https://chromium-review.googlesource.com/c/1299554
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>

[modify] https://crrev.com/4a6e603768d21161b37c19dfd7794848e06d475c/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android__dbg_.json
[modify] https://crrev.com/4a6e603768d21161b37c19dfd7794848e06d475c/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android.json
[modify] https://crrev.com/4a6e603768d21161b37c19dfd7794848e06d475c/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android__dbg__fail.json
[modify] https://crrev.com/4a6e603768d21161b37c19dfd7794848e06d475c/scripts/slave/README.recipes.md
[modify] https://crrev.com/4a6e603768d21161b37c19dfd7794848e06d475c/scripts/slave/recipes/swarming/deterministic_build.py
[modify] https://crrev.com/4a6e603768d21161b37c19dfd7794848e06d475c/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android_fail.json

The build cache works pretty well, compare:

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Deterministic%20Linux/17374
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Deterministic%20Linux/17365

runhooks is over 6 min faster (but the two rmtrees are a few seconds slower).


On Windows, most hooks are cached, but the MSVC bit doesn't work yet, see e.g. the first runhooks on https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Windows%20deterministic/10519 :

________ running 'C:\b\swarming\w\ir\cache\vpython\15e091\Scripts\python.exe src/build/download_nacl_toolchains.py --mode nacl_core_sdk sync --extract' in 'C:\b\swarming\w\ir\cache\builder'
________ running 'C:\b\swarming\w\ir\cache\vpython\15e091\Scripts\python.exe src/build/vs_toolchain.py update --force' in 'C:\b\swarming\w\ir\cache\builder'
Windows toolchain out of date or doesn't exist, updating (Pro)...
  current_hashes: 
  desired_hash: 3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c
Copying gs://chrome-wintoolchain/3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c.zip...
/ [0 files][    0.0 B/886.8 MiB]                                                
==> NOTE: You are downloading one or more large file(s), which would
run significantly faster if you enabled sliced object downloads. This
feature is enabled by default but requires that compiled crcmod be
installed (see "gsutil help crcmod").

-
- [0 files][ 75.0 MiB/886.8 MiB]                                                
\
|
| [0 files][233.6 MiB/886.8 MiB]                                                
/
/ [0 files][380.5 MiB/886.8 MiB]                                                
-
\
\ [0 files][535.0 MiB/886.8 MiB]                                                
|
/
/ [0 files][688.1 MiB/886.8 MiB]                                                
-
- [0 files][841.5 MiB/886.8 MiB]                                                
\
\ [1 files][886.8 MiB/886.8 MiB]                                                

Operation completed over 1 objects/886.8 MiB.                                    
Extracting c:\b\swarming\w\ir\tmp\t\tmp7xvphj\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c.zip...
Calculating hash of toolchain in vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c. Please wait...

(which takes over a minute)

The env:
 cwd: 'C:\\b\\swarming\\w\\ir\\cache\\builder'
 'DEPOT_TOOLS_WIN_TOOLCHAIN_ROOT': 'C:\\b\\swarming\\w\\ir\\cache\\win_toolchain',


Either there's a typo in the depot_tools env var, or its value is in a place that gets clobbered, or something.


Also, for some reason on Windows, the "first build" got a lot slower. It used to consistently be a pretty zippy 20 min, but on the first build after the change it was 1h20m instead and now ~45 min twice (
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Windows%20deterministic/10517
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Windows%20deterministic/10518
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Windows%20deterministic/10519
). I don't understand why that would change. Isn't goma's cache supposed to be build directory independent?

tikuta: If two compiles have different environment variables, can they share a goma cache?

Comparing these:
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8931679832530451536/+/steps/First_build/0/stdout
https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8931718183077012992/+/steps/First_build/0/stdout

the main difference is that the slow one has

 env: {'CHROME_HEADLESS': '1',
 'DEPOT_TOOLS_WIN_TOOLCHAIN_ROOT': 'C:\\b\\swarming\\w\\ir\\cache\\win_toolchain',
 'GOMA_USE_LOCAL': 'false'}

while the fast one has

 env: {'CHROME_HEADLESS': '1', 'GOMA_USE_LOCAL': 'false'}


Does the presence of DEPOT_TOOLS_WIN_TOOLCHAIN_ROOT prevent sharing the goma cache? (The regular chrome builds don't set that env var at all, I don't know how they don't blow away their msvc all the time. But we don't have to set it on these bots during the compile step if that would help.)

I think https://chromium.googlesource.com/chromium/tools/build/+/110ff4ba13c7b7354c3e6cb8b56eab856b3e523e probably answers my goma question.

For "how does this work on the real bots", I think the answer is that build/vs_toolchain uses find_depot_tools from https://cs.chromium.org/chromium/src/build/find_depot_tools.py?q=find_depot_tools&sq=package:chromium&dr which returns src/third_party/depot_tools if that's DEPS'd in at that location (like it is in chromium builds), unless DEPOT_TOOLS_WIN_TOOLCHAIN_ROOT is set. So if depot_tools is deps'd in, one should _not_ set DEPOT_TOOLS_WIN_TOOLCHAIN_ROOT. This will help the goma cache, and since the build dir is now in a cache that should also prevent re-downloads. angle doesn't deps in depot_tools, which is why it needed this env var in https://bugs.chromium.org/p/angleproject/issues/detail?id=2793 (+jmadill: Just FYI, no action needed.)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/ca671f463540505c136374cd4d1412448e563475

commit ca671f463540505c136374cd4d1412448e563475
Author: Nico Weber <thakis@chromium.org>
Date: Thu Oct 25 21:23:20 2018

Remove DEPOT_TOOLS_WIN_TOOLCHAIN_ROOT on deterministic bots.

Looks like it hurts more than it helps on regular chromium bots: The normal
builder cache should keep msvc cached, and setting a custom value prevents
goma cache reuse.

TBR=tandrii
Bug:  897854 
Change-Id: I608a86198ed853a7f7c643288130f8df319a7000
Reviewed-on: https://chromium-review.googlesource.com/c/1300233
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>

[modify] https://crrev.com/ca671f463540505c136374cd4d1412448e563475/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic_fail.json
[modify] https://crrev.com/ca671f463540505c136374cd4d1412448e563475/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic.json
[modify] https://crrev.com/ca671f463540505c136374cd4d1412448e563475/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic_fail.json
[modify] https://crrev.com/ca671f463540505c136374cd4d1412448e563475/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic.json
[modify] https://crrev.com/ca671f463540505c136374cd4d1412448e563475/scripts/slave/README.recipes.md
[modify] https://crrev.com/ca671f463540505c136374cd4d1412448e563475/scripts/slave/recipes/swarming/deterministic_build.py

Stephen has more knowledge how it works for chromium than me.

I believe I have everything figured out by now, thanks :-) If the build following https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Windows%20deterministic/10520 doesn't re-download msvc I'm all set.

Seems fixed?

Yes!