New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 876915 link

Starred by 1 user
Status: Fixed
Owner: ----
Closed: Sep 7
Cc:
█ dpranke@google.com
tandrii@chromium.org
tikuta@chromium.org
maruel@google.com
thakis@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug

Blocking:
issue 330263



Sign in to add a comment

Deterministic builds should correctly generate .isolated files

Project Member Reported by erikc...@chromium.org, Aug 22 
The recipe for Linux Deterministic build attempts to:
  1) Generate .isolated files
  2) Compare the results

(1) has been failing for quite some time [I don't know when this started]. Looking at a successful build: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Deterministic%20Linux%20%28dbg%29/2462, we see that no isolate targets are found and no .isolated files are generated.

I recently updated the recipe to generate .isolated.gen.json files:
https://chromium-review.googlesource.com/c/chromium/tools/build/+/1180706

This then causes (1) to at least trigger, but that in turn causes actual errors, since some of the .isolate files generated reference non-existence files and directories.  [e.g. /media_router_e2e_tests.isolate]. This happens because these recipes are normally used in special downstream checkouts [I assume, but they're broken and never used (?)].

In the short term, I'm just going to disable (1) because it was already doing nothing, so that's a no-op.

Comment 1 by thakis@chromium.org, Aug 22 

Can we put the isolates that reference non-existent things behind a gn arg conditional so that they have to be explicitly opted in? Then if they are actually needed downstream somewhere, that downstream can set the gn flag, and if nobody sets the flag for some of these isolates we know it's unused and we can delete it.

Comment 2 by erikc...@chromium.org, Aug 22 

As a side note, (2) will stop working too once I land the CL, since we no longer compare .isolated files, only files referenced by .isolate files. I'll fix that shortly.

Comment 3 by dpranke@chromium.org, Aug 22 

What thakis suggests sounds like the right fix; it basically amounts to "fix the build", of course.
Project Member

Comment 4 by bugdroid1@chromium.org, Aug 23 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3725410415f3611528ba4900eb82b8f5ea9ab615

commit 3725410415f3611528ba4900eb82b8f5ea9ab615
Author: Erik Chen <erikchen@chromium.org>
Date: Thu Aug 23 00:16:28 2018

compare_build_artifacts.py should compare .isolated files

The option --use-isolate-files causes compare_build_artifacts.py to
compare the files referenced by .isolate. It should also compare the
contents of .isolated files.

Bug:  876915 
Change-Id: I06eb0d3373493db90b75befe9e0b47c9de58cbe0
Reviewed-on: https://chromium-review.googlesource.com/1185991
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Erik Chen <erikchen@chromium.org>
Cr-Commit-Position: refs/heads/master@{#585330}
[modify] https://crrev.com/3725410415f3611528ba4900eb82b8f5ea9ab615/tools/determinism/compare_build_artifacts.py
Project Member

Comment 5 by bugdroid1@chromium.org, Aug 23 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f9e001f4013c1694abff87cb23f07c1e4c4a01f1

commit f9e001f4013c1694abff87cb23f07c1e4c4a01f1
Author: Erik Chen <erikchen@chromium.org>
Date: Thu Aug 23 01:37:11 2018

Prevent downstream media targets from appearing in upstream chromium builds.

This CL adds a gn arg enable_downstream_media_tests, which downstream consumers
of chromium can set to make these builds appear. This prevents building and
isolate generation [which fails] in upstream chromium builds that try to
build/isolate all targets.

Bug:  876915 
Change-Id: Ic0f892680e889a5cd5d92b8853800a3c08926a9b
Reviewed-on: https://chromium-review.googlesource.com/1186102
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Erik Chen <erikchen@chromium.org>
Cr-Commit-Position: refs/heads/master@{#585357}
[modify] https://crrev.com/f9e001f4013c1694abff87cb23f07c1e4c4a01f1/chrome/test/media_router/BUILD.gn
Project Member

Comment 6 by bugdroid1@chromium.org, Aug 28 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8

commit 5f93eb231dd5cd4032f160bb56c2a46e3541e2f8
Author: erikchen <erikchen@chromium.org>
Date: Tue Aug 28 17:46:52 2018

[Reland #1] Update deterministic build to compare artifacts based on .isolate files.

The first attempt to land this CL ran into a bug with .isolates referencing
non-existent files. This caused isolate generation to fail.

That problem has since been fixed:
https://chromium-review.googlesource.com/c/chromium/src/+/1186102

> This CL updates the deterministic build recipe to check that outputs referenced
> by .isolate files are identical, rather than just checking that certain
> executable files are identical. This will catch non-determinism in data/ GN
> dependencies, among other problems.
>
> This CL:
>   * Calls `mb.py --isolate-everything` for each of the two builds. This
>     generates .isolate files [but does not actually communicate with the isolate
>     server to create .isolated files].
>   * Adds the flag `--use-isolate-files` to arguments passed to
>     compare_build_artifacts.py. This instructs the script to compare all files
>     specified by .isolate files.
>
> Bug:  870731 
> Change-Id: I9179e93c12c4bc1879224a4d198f7916402b386e
> Reviewed-on: https://chromium-review.googlesource.com/1180706
> Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
> Reviewed-by: Nico Weber <thakis@chromium.org>
> Commit-Queue: Erik Chen <erikchen@chromium.org>

Bug:  870731 ,  876915 
Change-Id: Iccd95a046b5a0431a6f1ce3dff148102c862f864
Reviewed-on: https://chromium-review.googlesource.com/1185911
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Erik Chen <erikchen@chromium.org>

[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic__dbg_.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipe_modules/chromium/api.py
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg_.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android__dbg__fail.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic_fail.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipe_modules/isolate/tests/compare_build_artifacts.expected/basic.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic_fail.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic_fail.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android__dbg_.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android_fail.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.py
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic__dbg__fail.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux_fail.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg__fail.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic_fail.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/README.recipes.md
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipe_modules/isolate/api.py
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipe_modules/isolate/tests/compare_build_artifacts.expected/failure.json
Project Member

Comment 7 by bugdroid1@chromium.org, Aug 28 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b5d68e39ad37003842d10e0e7c06603ac5058883

commit b5d68e39ad37003842d10e0e7c06603ac5058883
Author: erikchen <erikchen@chromium.org>
Date: Tue Aug 28 20:04:29 2018

Update deterministic build whitelist for Linux.

browser_tests.isolated is not deterministic on Linux. This was previously not
noticed because the .isolated file was not being generated.

Bug:  876915 ,  870731 ,  330263 
Change-Id: I3022bc0b6e28478b44df54979c5bb5e5fa9cc834
Reviewed-on: https://chromium-review.googlesource.com/1194469
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Erik Chen <erikchen@chromium.org>
Cr-Commit-Position: refs/heads/master@{#586823}
[modify] https://crrev.com/b5d68e39ad37003842d10e0e7c06603ac5058883/tools/determinism/deterministic_build_whitelist.pyl

Comment 8 by dtapu...@chromium.org, Aug 29

Components: Build

Comment 9 by thakis@chromium.org, Sep 7

Blocking: 330263
Is this done?

Comment 10 by erikc...@chromium.org, Sep 7

Status: Fixed (was: Untriaged)
Project Member

Comment 11 by bugdroid1@chromium.org, Nov 22 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/00e5353103a6dca9622e819e48138152fded70fc

commit 00e5353103a6dca9622e819e48138152fded70fc
Author: Nico Weber <thakis@chromium.org>
Date: Thu Nov 22 14:33:14 2018

Deterministic bots: Stop running api.isolate.isolate_tests().

Now that we're on gn, mb writes the .isolate file which lists all files in it,
and isolate_tests() then takes that .isolate file to create a .isolated file
containing hashes of all binaries, and names of hidden .tar archives that
have content-addressed names with all the contents of these files.

Creating the .isolated files is pretty slow (all build outputs are be
zipped up), the .isolated files contain the name of the build dir which
means it must be filtered out, and since the tar file names are content-addressed
those would have to be filtered out at compare time too, and the .isolated
files don't buy us anything we don't already get from the .isolate files
(compare_build_artifacts.py gets its file list from the .isolate files, not
from the .isolated files).

So stop running this step. It will make the bots cycle faster, and until
.isolated files are builddir-name independent it allows removing some
workarounds from compare_build_artifacts.py.

Bug: 907488,899438, 876915 
Change-Id: I416d5966755a1d199842bcdd94b80aed156b5594
Reviewed-on: https://chromium-review.googlesource.com/c/1347346
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Hans Wennborg <hans@chromium.org>

[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic_fail.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic__dbg_.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux_fail.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg__fail.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic_fail.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg_.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic_fail.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic__dbg__fail.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic_fail.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.py
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic.json
Project Member

Comment 12 by bugdroid1@chromium.org, Nov 26 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bb6843b1270f1692f29e948bd57ab2e5b7c9a747

commit bb6843b1270f1692f29e948bd57ab2e5b7c9a747
Author: Nico Weber <thakis@chromium.org>
Date: Mon Nov 26 19:11:38 2018

Stop listing .isolated files in deterministic build whitelist.

Unused after https://chromium-review.googlesource.com/c/chromium/tools/build/+/1347346

Also remove code that's been unused since that change related to .isolated
files from compare_build_artifacts.py

Bug: 899438, 876915 
Change-Id: Ie572c2a27924b72a2ffb86165afca37fcbd667d6
Reviewed-on: https://chromium-review.googlesource.com/c/1351378
Reviewed-by: Erik Chen <erikchen@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#610906}
[modify] https://crrev.com/bb6843b1270f1692f29e948bd57ab2e5b7c9a747/tools/determinism/compare_build_artifacts.py
[modify] https://crrev.com/bb6843b1270f1692f29e948bd57ab2e5b7c9a747/tools/determinism/deterministic_build_whitelist.pyl
►

Sign in to add a comment