New issue
Advanced search Search tips

Issue 870731 link

Starred by 2 users
Status: Fixed
Owner:
erikc...@chromium.org
Closed: Sep 7
Cc:
tikuta@chromium.org
yyanagisawa@chromium.org
thakis@chromium.org
mar...@chromium.org
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug

Blocked on:
issue 875279
Blocking:
issue 330263



Sign in to add a comment

Deterministic builds should check that all build artifacts are identical.

Project Member Reported by erikc...@chromium.org, Aug 3 
The current script: https://cs.chromium.org/chromium/src/tools/determinism/compare_build_artifacts.py

Only checks that a subset of artifacts [executable artifacts with specific file extensions] are equivalent. This caused us to miss non-determinism added from a data dependency:

https://bugs.chromium.org/p/chromium/issues/detail?id=870622

I spoke with maruel@, who mentioned that the current script was just intended as a starting point.

Comment 1 by tikuta@chromium.org, Aug 3

Cc: tikuta@chromium.org yyanagisawa@chromium.org

Comment 2 by erikc...@chromium.org, Aug 8 

Here's a "diff -rq" output from two full builds on Linux [same commit, same output directory]:

"""
erikchen@erikchen:~/projects/chromium-linux/src$ diff -rq out.2/gn out.3/gn
Files out.2/gn/args.gn and out.3/gn/args.gn differ
Files out.2/gn/gen/base/bind_unittest_nc.cc.log and out.3/gn/gen/base/bind_unittest_nc.cc.log differ
Files out.2/gn/gen/base/callback_list_unittest_nc.cc.log and out.3/gn/gen/base/callback_list_unittest_nc.cc.log differ
Files out.2/gn/gen/base/callback_unittest_nc.cc.log and out.3/gn/gen/base/callback_unittest_nc.cc.log differ
Files out.2/gn/gen/base/field_trial_params_unittest_nc.cc.log and out.3/gn/gen/base/field_trial_params_unittest_nc.cc.log differ
Files out.2/gn/gen/base/histogram_unittest_nc.cc.log and out.3/gn/gen/base/histogram_unittest_nc.cc.log differ
Files out.2/gn/gen/base/optional_unittest_nc.cc.log and out.3/gn/gen/base/optional_unittest_nc.cc.log differ
Files out.2/gn/gen/base/ref_counted_unittest_nc.cc.log and out.3/gn/gen/base/ref_counted_unittest_nc.cc.log differ
Files out.2/gn/gen/base/span_unittest_nc.cc.log and out.3/gn/gen/base/span_unittest_nc.cc.log differ
Files out.2/gn/gen/base/string16_unittest_nc.cc.log and out.3/gn/gen/base/string16_unittest_nc.cc.log differ
Files out.2/gn/gen/base/task_traits_unittest_nc.cc.log and out.3/gn/gen/base/task_traits_unittest_nc.cc.log differ
Files out.2/gn/gen/base/thread_annotations_unittest_nc.cc.log and out.3/gn/gen/base/thread_annotations_unittest_nc.cc.log differ
Files out.2/gn/gen/base/weak_ptr_unittest_nc.cc.log and out.3/gn/gen/base/weak_ptr_unittest_nc.cc.log differ
Files out.2/gn/gen/chrome/test/chromedriver/chrome/embedded_automation_extension.cc and out.3/gn/gen/chrome/test/chromedriver/chrome/embedded_automation_extension.cc differ
Files out.2/gn/gen/chrome/test/chromedriver/chrome/embedded_automation_extension.h and out.3/gn/gen/chrome/test/chromedriver/chrome/embedded_automation_extension.h differ
Files out.2/gn/gen/chrome/test/chromedriver/chrome/js.cc and out.3/gn/gen/chrome/test/chromedriver/chrome/js.cc differ
Files out.2/gn/gen/chrome/test/chromedriver/chrome/js.h and out.3/gn/gen/chrome/test/chromedriver/chrome/js.h differ
Files out.2/gn/gen/chrome/test/chromedriver/chrome/user_data_dir.cc and out.3/gn/gen/chrome/test/chromedriver/chrome/user_data_dir.cc differ
Files out.2/gn/gen/chrome/test/chromedriver/chrome/user_data_dir.h and out.3/gn/gen/chrome/test/chromedriver/chrome/user_data_dir.h differ
Files out.2/gn/gen/chrome/test/chromedriver/version.cc and out.3/gn/gen/chrome/test/chromedriver/version.cc differ
Files out.2/gn/gen/chrome/test/chromedriver/version.h and out.3/gn/gen/chrome/test/chromedriver/version.h differ
Files out.2/gn/gen/third_party/blink/renderer/bindings/core/ComponentInfoCore.pickle and out.3/gn/gen/third_party/blink/renderer/bindings/core/ComponentInfoCore.pickle differ
Files out.2/gn/gen/third_party/blink/renderer/bindings/modules/ComponentInfoModules.pickle and out.3/gn/gen/third_party/blink/renderer/bindings/modules/ComponentInfoModules.pickle differ
Files out.2/gn/gen/third_party/blink/renderer/bindings/scripts/lextab.pyc and out.3/gn/gen/third_party/blink/renderer/bindings/scripts/lextab.pyc differ
Files out.2/gn/.ninja_deps and out.3/gn/.ninja_deps differ
Files out.2/gn/.ninja_log and out.3/gn/.ninja_log differ
Files out.2/gn/pyproto/chrome/browser/ssl/ssl_error_assistant_pb2.pyc and out.3/gn/pyproto/chrome/browser/ssl/ssl_error_assistant_pb2.pyc differ
Files out.2/gn/pyproto/chrome/common/safe_browsing/download_file_types_pb2.pyc and out.3/gn/pyproto/chrome/common/safe_browsing/download_file_types_pb2.pyc differ
Files out.2/gn/remoting-webapp.v2.zip and out.3/gn/remoting-webapp.v2.zip differ
Files out.2/gn/v8_context_snapshot.bin and out.3/gn/v8_context_snapshot.bin differ
Files out.2/gn/zucchini_apply_fuzzer_seed_corpus.zip and out.3/gn/zucchini_apply_fuzzer_seed_corpus.zip differ
"""

Comment 3 by thakis@chromium.org, Aug 8 

Is that all files referenced from out*/gn/*.isolate or all files? (See https://bugs.chromium.org/p/chromium/issues/detail?id=869348#c21)

Comment 4 by erikc...@chromium.org, Aug 14 

This was all files. 

I guess it would make more sense to just check files listed in *.isolate, which means we need to generate the *.isolate files -- I guess we'd have to parse gn_isolate_map.pyl and figure out the appropriate targets. We probably also want to add a dry-run option to mb.py. 

Is there a better way to do this?

Comment 5 by mar...@chromium.org, Aug 14 

Not afaik.

Comment 6 by erikc...@chromium.org, Aug 15

Owner: erikc...@chromium.org
Status: Assigned (was: Untriaged)
Project Member

Comment 7 by bugdroid1@chromium.org, Aug 15 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/01ab8973a37aa937c132decc02dc103fd99e3e14

commit 01ab8973a37aa937c132decc02dc103fd99e3e14
Author: erikchen <erikchen@chromium.org>
Date: Wed Aug 15 19:24:49 2018

Add a --use-isolate-files option to compare_build_artifacts.py.

Previously, compare_build_artifacts.py used some basic heuristics to figure out
which files to compare. This new option instead scans the .isolate files in the
output directories, and uses their contents to determine which files to compare.

Change-Id: I78bba7d3c06be682da694eeeb0dbe195fe312ace
Bug:  870731 
Reviewed-on: https://chromium-review.googlesource.com/1175043
Commit-Queue: Erik Chen <erikchen@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Cr-Commit-Position: refs/heads/master@{#583352}
[modify] https://crrev.com/01ab8973a37aa937c132decc02dc103fd99e3e14/tools/determinism/compare_build_artifacts.py
Project Member

Comment 8 by bugdroid1@chromium.org, Aug 18 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3033b341d7e08a9d4f7923c96f3d69662ad1ff7d

commit 3033b341d7e08a9d4f7923c96f3d69662ad1ff7d
Author: erikchen <erikchen@chromium.org>
Date: Sat Aug 18 00:54:35 2018

Add isolate-everything command to mb.py

The command generates a .isolate for every compatible gn target.

This will be used by the deterministic builder to check that all isolates
dependencies are identical between two builds.

Change-Id: I4f6a515e49f9cc207dd2f54c796531d12dfeb771
Bug:  870731 
Reviewed-on: https://chromium-review.googlesource.com/1176441
Commit-Queue: Erik Chen <erikchen@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#584268}
[modify] https://crrev.com/3033b341d7e08a9d4f7923c96f3d69662ad1ff7d/tools/mb/mb.py
Project Member

Comment 9 by bugdroid1@chromium.org, Aug 20 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/87f71a51f921af191478798a33a47ee392ff3c01

commit 87f71a51f921af191478798a33a47ee392ff3c01
Author: Benoit L <lizeb@chromium.org>
Date: Mon Aug 20 09:38:00 2018

Revert "Add isolate-everything command to mb.py"

This reverts commit 3033b341d7e08a9d4f7923c96f3d69662ad1ff7d.

Reason for revert: Suspected culprit for crbug.com/875713 (broken official bots).

Original change's description:
> Add isolate-everything command to mb.py
> 
> The command generates a .isolate for every compatible gn target.
> 
> This will be used by the deterministic builder to check that all isolates
> dependencies are identical between two builds.
> 
> Change-Id: I4f6a515e49f9cc207dd2f54c796531d12dfeb771
> Bug:  870731 
> Reviewed-on: https://chromium-review.googlesource.com/1176441
> Commit-Queue: Erik Chen <erikchen@chromium.org>
> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#584268}

TBR=maruel@chromium.org,dpranke@chromium.org,erikchen@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  870731 , 875713
Change-Id: I5110467272abbffc96bdfa8a11261822dd96a065
Reviewed-on: https://chromium-review.googlesource.com/1179862
Reviewed-by: Benoit L <lizeb@chromium.org>
Commit-Queue: Benoit L <lizeb@chromium.org>
Cr-Commit-Position: refs/heads/master@{#584389}
[modify] https://crrev.com/87f71a51f921af191478798a33a47ee392ff3c01/tools/mb/mb.py
Project Member

Comment 10 by bugdroid1@chromium.org, Aug 20

Labels: merge-merged-3528
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5fea3822808267ef92f6fee7fd6891fa94aaeb9f

commit 5fea3822808267ef92f6fee7fd6891fa94aaeb9f
Author: Benoit L <lizeb@chromium.org>
Date: Mon Aug 20 09:53:01 2018

Revert "Add isolate-everything command to mb.py"

This reverts commit 3033b341d7e08a9d4f7923c96f3d69662ad1ff7d.

Reason for revert: Suspected culprit for crbug.com/875713 (broken official bots).

Original change's description:
> Add isolate-everything command to mb.py
> 
> The command generates a .isolate for every compatible gn target.
> 
> This will be used by the deterministic builder to check that all isolates
> dependencies are identical between two builds.
> 
> Change-Id: I4f6a515e49f9cc207dd2f54c796531d12dfeb771
> Bug:  870731 
> Reviewed-on: https://chromium-review.googlesource.com/1176441
> Commit-Queue: Erik Chen <erikchen@chromium.org>
> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#584268}

TBR=maruel@chromium.org,dpranke@chromium.org,erikchen@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  870731 , 875713
Change-Id: I5110467272abbffc96bdfa8a11261822dd96a065
Reviewed-on: https://chromium-review.googlesource.com/1179862
Reviewed-by: Benoit L <lizeb@chromium.org>
Commit-Queue: Benoit L <lizeb@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#584389}(cherry picked from commit 87f71a51f921af191478798a33a47ee392ff3c01)
Reviewed-on: https://chromium-review.googlesource.com/1180981
Cr-Commit-Position: refs/branch-heads/3528@{#3}
Cr-Branched-From: 2d67fa3c43ccc16edae0a742d902dd437795959b-refs/heads/master@{#584349}
[modify] https://crrev.com/5fea3822808267ef92f6fee7fd6891fa94aaeb9f/tools/mb/mb.py
Project Member

Comment 11 by bugdroid1@chromium.org, Aug 21 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/42df41de254bbda893f7bdf80bd1a5b086f6fffc

commit 42df41de254bbda893f7bdf80bd1a5b086f6fffc
Author: Erik Chen <erikchen@chromium.org>
Date: Tue Aug 21 17:13:31 2018

[Reland #1] Add isolate-everything command to mb.py

The first attempt to land this CL failed because there is one consumer that
attempts to isolate a target called 'official_tests' which also has type
'additional_compile_target'. This is a misuse of the unofficial API for mb.py,
but is not trivial to fix.

This CL simply marks this as a special case a leaves a TODO for the responsible
party.

> The command generates a .isolate for every compatible gn target.
>
> This will be used by the deterministic builder to check that all isolates
> dependencies are identical between two builds.
>
> Bug:  870731 
> Reviewed-on: https://chromium-review.googlesource.com/1176441
> Commit-Queue: Erik Chen <erikchen@chromium.org>
> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#584268}

Change-Id: I1d31870b0513911b3ff6ebe296ea812b9036bf45
Bug: 876065, 875713,  870731 
Reviewed-on: https://chromium-review.googlesource.com/1182621
Commit-Queue: Erik Chen <erikchen@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#584800}
[modify] https://crrev.com/42df41de254bbda893f7bdf80bd1a5b086f6fffc/tools/mb/mb.py
Project Member

Comment 12 by bugdroid1@chromium.org, Aug 22 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9b3e5ba4d3b074cf32c87bb769750b22b11e56c1

commit 9b3e5ba4d3b074cf32c87bb769750b22b11e56c1
Author: erikchen <erikchen@chromium.org>
Date: Wed Aug 22 00:40:36 2018

Gracefully handle missing files in compare_build_artifacts.

If files are specified by a .isolate and missing, this should be handled like
any other error in compare_build_artifacts. It should not throw an exception.

Bug:  870731 
Change-Id: If29a180539ffa148d5788ac2ecd626205e2eb090
Reviewed-on: https://chromium-review.googlesource.com/1183998
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Erik Chen <erikchen@chromium.org>
Cr-Commit-Position: refs/heads/master@{#584939}
[modify] https://crrev.com/9b3e5ba4d3b074cf32c87bb769750b22b11e56c1/tools/determinism/compare_build_artifacts.py
Project Member

Comment 13 by bugdroid1@chromium.org, Aug 22 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d3aaaf5c3915d25ec9942f793176abb1a6809605

commit d3aaaf5c3915d25ec9942f793176abb1a6809605
Author: erikchen <erikchen@chromium.org>
Date: Wed Aug 22 00:41:33 2018

Add deterministic build whitelist files for Linux.

Previously, compare_build_artifacts.py only compared certain executables in the
output directories. This CL:
https://chromium-review.googlesource.com/c/chromium/tools/build/+/1180706 makes
deterministic builder use .isolate files to compare both data files and the
appropriate executables. With this change, we've discovered that the data files
included in this CL are also not deterministic.

Bug:  870731 ,  330263 
Change-Id: Ia02c3f99d35ab690f5a709c0adb987d2820201b5
Reviewed-on: https://chromium-review.googlesource.com/1183999
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Erik Chen <erikchen@chromium.org>
Cr-Commit-Position: refs/heads/master@{#584940}
[modify] https://crrev.com/d3aaaf5c3915d25ec9942f793176abb1a6809605/tools/determinism/deterministic_build_whitelist.pyl
Project Member

Comment 14 by bugdroid1@chromium.org, Aug 22 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc

commit 9e6abc1d2450b4d764063e73132fa0db8d6cc2cc
Author: Erik Chen <erikchen@chromium.org>
Date: Wed Aug 22 16:30:21 2018

Update deterministic build to compare artifacts based on .isolate files.

This CL updates the deterministic build recipe to check that outputs referenced
by .isolate files are identical, rather than just checking that certain
executable files are identical. This will catch non-determinism in data/ GN
dependencies, among other problems.

This CL:
  * Calls `mb.py --isolate-everything` for each of the two builds. This
    generates .isolate files [but does not actually communicate with the isolate
    server to create .isolated files].
  * Adds the flag `--use-isolate-files` to arguments passed to
    compare_build_artifacts.py. This instructs the script to compare all files
    specified by .isolate files.

Bug:  870731 
Change-Id: I9179e93c12c4bc1879224a4d198f7916402b386e
Reviewed-on: https://chromium-review.googlesource.com/1180706
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Erik Chen <erikchen@chromium.org>

[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic__dbg_.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipe_modules/chromium/api.py
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg_.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android__dbg__fail.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic_fail.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipe_modules/isolate/tests/compare_build_artifacts.expected/basic.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic_fail.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic_fail.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android__dbg_.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android_fail.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.py
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic__dbg__fail.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux_fail.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg__fail.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic_fail.json
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/README.recipes.md
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipe_modules/isolate/api.py
[modify] https://crrev.com/9e6abc1d2450b4d764063e73132fa0db8d6cc2cc/scripts/slave/recipe_modules/isolate/tests/compare_build_artifacts.expected/failure.json
Project Member

Comment 15 by bugdroid1@chromium.org, Aug 22 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/ce4dcdef107f316ed4ecf188a91c6557c687fce7

commit ce4dcdef107f316ed4ecf188a91c6557c687fce7
Author: Erik Chen <erikchen@chromium.org>
Date: Wed Aug 22 19:55:05 2018

Revert "Update deterministic build to compare artifacts based on .isolate files."

This reverts commit 9e6abc1d2450b4d764063e73132fa0db8d6cc2cc.

Caused failures on Linux Deterministic Builder [DBG + release]

https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8937471910311557792/+/steps/isolate_tests/0/stdout

https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8937468729791837408/+/steps/isolate_tests/0/stdout

Original change's description:
> Update deterministic build to compare artifacts based on .isolate files.
> 
> This CL updates the deterministic build recipe to check that outputs referenced
> by .isolate files are identical, rather than just checking that certain
> executable files are identical. This will catch non-determinism in data/ GN
> dependencies, among other problems.
> 
> This CL:
>   * Calls `mb.py --isolate-everything` for each of the two builds. This
>     generates .isolate files [but does not actually communicate with the isolate
>     server to create .isolated files].
>   * Adds the flag `--use-isolate-files` to arguments passed to
>     compare_build_artifacts.py. This instructs the script to compare all files
>     specified by .isolate files.
> 
> Bug:  870731 
> Change-Id: I9179e93c12c4bc1879224a4d198f7916402b386e
> Reviewed-on: https://chromium-review.googlesource.com/1180706
> Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
> Reviewed-by: Nico Weber <thakis@chromium.org>
> Commit-Queue: Erik Chen <erikchen@chromium.org>

TBR=thakis@chromium.org,dpranke@chromium.org,erikchen@chromium.org,tikuta@chromium.org

Change-Id: I79b6fab0a3240f7ad422d5adcd21e65427a75481
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  870731 
Reviewed-on: https://chromium-review.googlesource.com/1185701
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Erik Chen <erikchen@chromium.org>

[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic__dbg_.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipe_modules/chromium/api.py
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg_.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android__dbg__fail.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic_fail.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipe_modules/isolate/tests/compare_build_artifacts.expected/basic.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic_fail.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic_fail.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android__dbg_.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android_fail.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.py
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic__dbg__fail.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux_fail.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg__fail.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic_fail.json
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/README.recipes.md
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipe_modules/isolate/api.py
[modify] https://crrev.com/ce4dcdef107f316ed4ecf188a91c6557c687fce7/scripts/slave/recipe_modules/isolate/tests/compare_build_artifacts.expected/failure.json
Project Member

Comment 16 by bugdroid1@chromium.org, Aug 28 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8

commit 5f93eb231dd5cd4032f160bb56c2a46e3541e2f8
Author: erikchen <erikchen@chromium.org>
Date: Tue Aug 28 17:46:52 2018

[Reland #1] Update deterministic build to compare artifacts based on .isolate files.

The first attempt to land this CL ran into a bug with .isolates referencing
non-existent files. This caused isolate generation to fail.

That problem has since been fixed:
https://chromium-review.googlesource.com/c/chromium/src/+/1186102

> This CL updates the deterministic build recipe to check that outputs referenced
> by .isolate files are identical, rather than just checking that certain
> executable files are identical. This will catch non-determinism in data/ GN
> dependencies, among other problems.
>
> This CL:
>   * Calls `mb.py --isolate-everything` for each of the two builds. This
>     generates .isolate files [but does not actually communicate with the isolate
>     server to create .isolated files].
>   * Adds the flag `--use-isolate-files` to arguments passed to
>     compare_build_artifacts.py. This instructs the script to compare all files
>     specified by .isolate files.
>
> Bug:  870731 
> Change-Id: I9179e93c12c4bc1879224a4d198f7916402b386e
> Reviewed-on: https://chromium-review.googlesource.com/1180706
> Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
> Reviewed-by: Nico Weber <thakis@chromium.org>
> Commit-Queue: Erik Chen <erikchen@chromium.org>

Bug:  870731 ,  876915 
Change-Id: Iccd95a046b5a0431a6f1ce3dff148102c862f864
Reviewed-on: https://chromium-review.googlesource.com/1185911
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Erik Chen <erikchen@chromium.org>

[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic__dbg_.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipe_modules/chromium/api.py
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg_.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android__dbg__fail.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic_fail.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipe_modules/isolate/tests/compare_build_artifacts.expected/basic.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic_fail.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic_fail.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android__dbg_.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android_fail.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.py
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic__dbg__fail.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux_fail.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg__fail.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic_fail.json
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/README.recipes.md
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipe_modules/isolate/api.py
[modify] https://crrev.com/5f93eb231dd5cd4032f160bb56c2a46e3541e2f8/scripts/slave/recipe_modules/isolate/tests/compare_build_artifacts.expected/failure.json
Project Member

Comment 17 by bugdroid1@chromium.org, Aug 28 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b5d68e39ad37003842d10e0e7c06603ac5058883

commit b5d68e39ad37003842d10e0e7c06603ac5058883
Author: erikchen <erikchen@chromium.org>
Date: Tue Aug 28 20:04:29 2018

Update deterministic build whitelist for Linux.

browser_tests.isolated is not deterministic on Linux. This was previously not
noticed because the .isolated file was not being generated.

Bug:  876915 ,  870731 ,  330263 
Change-Id: I3022bc0b6e28478b44df54979c5bb5e5fa9cc834
Reviewed-on: https://chromium-review.googlesource.com/1194469
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Erik Chen <erikchen@chromium.org>
Cr-Commit-Position: refs/heads/master@{#586823}
[modify] https://crrev.com/b5d68e39ad37003842d10e0e7c06603ac5058883/tools/determinism/deterministic_build_whitelist.pyl

Comment 18 by erikc...@chromium.org, Sep 7

Cc: thakis@chromium.org
Status: Fixed (was: Assigned)

Comment 19 by thakis@chromium.org, Sep 7

Blocking: 330263

Comment 20 by thakis@chromium.org, Sep 7

Blockedon: 875279
►

Sign in to add a comment