New issue
Advanced search Search tips

Issue 899438 link

Starred by 2 users
Status: Assigned
Owner:
thakis@chromium.org
Cc:
dpranke@chromium.org
mar...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 3
Type: Bug

Blocked on: View detail
issue 871574
issue 906741
issue 907488
issue 908463
issue 912946
issue 917355
issue 899437
issue 899439
issue 899442
issue 908474
issue 918882
issue 919437
Blocking:
issue 314403
issue 910465
issue 910699



Sign in to add a comment

Make deterministic bots use two different build dirs, instead of one twice

Project Member Reported by thakis@google.com, Oct 27 
Current, the deterministic bots do a clobber build, then move the build dir elsewhere, and then build again in the original dir.

It would be better if they did builds in two separate build dirs (whose names have different length), then they'd also check that the output is build-dir-name-independent.

Also, then one of the two build dirs wouldn't have to be a clobber dir, which probably helps with cycle time (and it'd check that incremental builds are also deterministic).

Comment 1 by thakis@google.com, Oct 27

Blockedon: 899439

Comment 2 by thakis@chromium.org, Oct 27

Blocking: 314403
Components: Build
Owner: thakis@chromium.org
Status: Untriaged (was: Unconfirmed)
Full compare output for content.dll:

C:\src\chrome\src>python tools\determinism\compare_build_artifacts.py --first-build-dir=out\gn --second-build-dir=out\gn2 -t win
Epoch: a4 be d3 5b
content.dll: DIFFERENT (unexpected): 399 out of 119305216 bytes are different (0.00%)
Equals:           0
Expected diffs:   0
Unexpected diffs: 1
Unexpected files with diffs:

  content.dll
Checking content.dll difference: (19110 deps)
  obj/third_party/perfetto/protos/perfetto/config/zero/data_source_config.pbzero.obj                                                                                      : 2901 out of 53275 bytes are different (5.45%)
  obj/third_party/perfetto/protos/perfetto/trace/ftrace/zero/ftrace_event.pbzero.obj                                                                                      : 180821 out of 1137439 bytes are different (15.90%)
  obj/mojo/public/js/resources/mojo_bindings_resources_map.obj                                                                                                            : different size: 1468 != 1469
  obj/skia/skia/GrRadialGradientLayout.obj                                                                                                                                : different size: 761521 != 761524
  obj/skia/skia/GrGLShaderStringBuilder.obj                                                                                                                               : different size: 934441 != 934438
  obj/third_party/crashpad/crashpad/util/util/capture_context_win.obj                                                                                                     : 119 out of 1078 bytes are different (11.04%)
  obj/third_party/crashpad/crashpad/util/util/safe_terminate_process.obj                                                                                                  : 121 out of 464 bytes are different (26.08%)
  obj/v8/v8_external_snapshot/embedded.obj                                                                                                                                : 19 out of 5107415 bytes are different (0.00%)
  obj/third_party/perfetto/protos/perfetto/trace/zero/trace_packet.pbzero.obj                                                                                             : 6635 out of 111173 bytes are different (5.97%)


The crashpad files are harmless, they're just there because ml64.exe can't set a fixed timestamp (but we don't run ml64 on goma and come link time nothing keeps these timestamps around). I'm filing blockers for the others.

Comment 3 by thakis@chromium.org, Oct 27

Blockedon: 899437

Comment 4 by thakis@chromium.org, Oct 28

Blockedon: 899442
Project Member

Comment 5 by bugdroid1@chromium.org, Oct 30 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/de2f0bfe2c61fe2a823640d2f5ffcada755cac59

commit de2f0bfe2c61fe2a823640d2f5ffcada755cac59
Author: Nico Weber <thakis@chromium.org>
Date: Tue Oct 30 10:21:53 2018

win: Add hacky ml.exe wrapper that makes ml's output deterministic after the fact.

Without this, chrome.exe.pdb contains two absolute paths (to the two obj files
built by ml64 for crashpad).

Bug: 762167,330260,899438
Change-Id: If4e0195b51f7d9ff70d890fe3a5302f5961c94d6
Reviewed-on: https://chromium-review.googlesource.com/c/1303887
Reviewed-by: Reid Kleckner <rnk@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#603852}
[modify] https://crrev.com/de2f0bfe2c61fe2a823640d2f5ffcada755cac59/build/toolchain/win/BUILD.gn
[add] https://crrev.com/de2f0bfe2c61fe2a823640d2f5ffcada755cac59/build/toolchain/win/ml.py
Project Member

Comment 6 by bugdroid1@chromium.org, Nov 19 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/82baec82928317042527aab213e8f278bab859dc

commit 82baec82928317042527aab213e8f278bab859dc
Author: Nico Weber <thakis@chromium.org>
Date: Mon Nov 19 16:34:26 2018

Make "Windows deterministic" bot use two distinct build directories.

Previously, it would build in one dir, move it elsewhere, build in the first
dir again, move that elsewhere too, and then compare the two dirs.

Now, it builds in two completely distinct build directories, so the bot
now checks that no absolute paths are in the build output.

The bot now also doesn't clobber the first build dir, to check that clobber
and incremental builds produce the same output.

Bug: 899438
Change-Id: I3631c8b7be69d651e1abcc71d96f08c4e749ee2c
Reviewed-on: https://chromium-review.googlesource.com/c/1341113
Reviewed-by: Hans Wennborg <hans@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>

[modify] https://crrev.com/82baec82928317042527aab213e8f278bab859dc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic_fail.json
[modify] https://crrev.com/82baec82928317042527aab213e8f278bab859dc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic.json
[modify] https://crrev.com/82baec82928317042527aab213e8f278bab859dc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic_fail.json
[modify] https://crrev.com/82baec82928317042527aab213e8f278bab859dc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic.json
[modify] https://crrev.com/82baec82928317042527aab213e8f278bab859dc/scripts/slave/recipes/swarming/deterministic_build.py
Project Member

Comment 7 by bugdroid1@chromium.org, Nov 19 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/33d770573e5832eb009b0cb4767ad9d228c5610f

commit 33d770573e5832eb009b0cb4767ad9d228c5610f
Author: Nico Weber <thakis@chromium.org>
Date: Mon Nov 19 17:40:40 2018

Keep deleting .1 directory on "Windows deterministic".

Else, on the first build after
https://chromium-review.googlesource.com/c/chromium/tools/build/+/1341113
we'll have three build directories (Release, Release.1, Release.2, with the
.1 one not being used and just taking up space).

TBR=hans@chromium.org

Bug: 899438
Change-Id: Ic6d92d401d8207cd9803b02cc29dce79b4eb8c20
Reviewed-on: https://chromium-review.googlesource.com/c/1342378
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>

[modify] https://crrev.com/33d770573e5832eb009b0cb4767ad9d228c5610f/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic_fail.json
[modify] https://crrev.com/33d770573e5832eb009b0cb4767ad9d228c5610f/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic.json
[modify] https://crrev.com/33d770573e5832eb009b0cb4767ad9d228c5610f/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic_fail.json
[modify] https://crrev.com/33d770573e5832eb009b0cb4767ad9d228c5610f/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic.json
[modify] https://crrev.com/33d770573e5832eb009b0cb4767ad9d228c5610f/scripts/slave/recipes/swarming/deterministic_build.py
Project Member

Comment 8 by bugdroid1@chromium.org, Nov 19 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/9021f96295976c95a5c7392d534d7515a86a09af

commit 9021f96295976c95a5c7392d534d7515a86a09af
Author: Nico Weber <thakis@chromium.org>
Date: Mon Nov 19 18:35:20 2018

Remove the clobber config on the deterministic bots.

There's already a manual clobber step for removing the build dirs after
they got moved to a new location.

For the Windows bot, this causes the "regular" build dir to not be clobbered.
For the other bots, this is currently a no-op.

TBR=hans
Bug: 899438
Change-Id: I41e0bd8b4854d14e76c3c0b2148db7f4935d5a32
Reviewed-on: https://chromium-review.googlesource.com/c/1342380
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>

[modify] https://crrev.com/9021f96295976c95a5c7392d534d7515a86a09af/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic.json
[modify] https://crrev.com/9021f96295976c95a5c7392d534d7515a86a09af/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic__dbg_.json
[modify] https://crrev.com/9021f96295976c95a5c7392d534d7515a86a09af/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux_fail.json
[modify] https://crrev.com/9021f96295976c95a5c7392d534d7515a86a09af/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic.json
[modify] https://crrev.com/9021f96295976c95a5c7392d534d7515a86a09af/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux.json
[modify] https://crrev.com/9021f96295976c95a5c7392d534d7515a86a09af/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg__fail.json
[modify] https://crrev.com/9021f96295976c95a5c7392d534d7515a86a09af/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic_fail.json
[modify] https://crrev.com/9021f96295976c95a5c7392d534d7515a86a09af/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg_.json
[modify] https://crrev.com/9021f96295976c95a5c7392d534d7515a86a09af/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic_fail.json
[modify] https://crrev.com/9021f96295976c95a5c7392d534d7515a86a09af/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic__dbg__fail.json
[modify] https://crrev.com/9021f96295976c95a5c7392d534d7515a86a09af/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic.json
[modify] https://crrev.com/9021f96295976c95a5c7392d534d7515a86a09af/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic_fail.json
[modify] https://crrev.com/9021f96295976c95a5c7392d534d7515a86a09af/scripts/slave/README.recipes.md
[modify] https://crrev.com/9021f96295976c95a5c7392d534d7515a86a09af/scripts/slave/recipes/swarming/deterministic_build.py
[modify] https://crrev.com/9021f96295976c95a5c7392d534d7515a86a09af/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic.json
[modify] https://crrev.com/9021f96295976c95a5c7392d534d7515a86a09af/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic_fail.json
Project Member

Comment 9 by bugdroid1@chromium.org, Nov 19 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/6e178d2300e1896a00abcde3f37425b418ad7ef2

commit 6e178d2300e1896a00abcde3f37425b418ad7ef2
Author: Nico Weber <thakis@chromium.org>
Date: Mon Nov 19 19:10:30 2018

Run runhooks just once on the deterministic bots.

I don't think it's necessary to runhooks once per build now that we're on gn.

No intended behavior change; will make the bots a tiny bit faster.

TBR=hans
Bug: 899438
Change-Id: I7067870e2d6f77c0ade22d0784ad6c3e2aefda74
Reviewed-on: https://chromium-review.googlesource.com/c/1342958
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>

[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic_fail.json
[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic__dbg_.json
[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux_fail.json
[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic.json
[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux.json
[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android__dbg_.json
[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg__fail.json
[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic_fail.json
[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg_.json
[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android.json
[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic__dbg__fail.json
[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android__dbg__fail.json
[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic_fail.json
[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic.json
[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic.json
[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Android_fail.json
[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic.json
[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic_fail.json
[modify] https://crrev.com/6e178d2300e1896a00abcde3f37425b418ad7ef2/scripts/slave/recipes/swarming/deterministic_build.py

Comment 10 by thakis@chromium.org, Nov 19

Blockedon: 906741
Project Member

Comment 11 by bugdroid1@chromium.org, Nov 19 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/db9e62be87a2d0b2e8380bddf5060466d24ccca3

commit db9e62be87a2d0b2e8380bddf5060466d24ccca3
Author: Nico Weber <thakis@chromium.org>
Date: Mon Nov 19 23:50:39 2018

Move "Windows deterministic" bot back to old build dir layout for now.

The .isolated files contain the name of the build dir, so havingb
build dirs out/Release and out/Release.2 makes the isolated comparison fail.
Need to come up with a plan for that before relanding.

TBR=hans@chromium.org

Bug: 899438
Change-Id: I058fd0db13eafe3f42307efd48d31db462261e6d
Reviewed-on: https://chromium-review.googlesource.com/c/1342960
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>

[modify] https://crrev.com/db9e62be87a2d0b2e8380bddf5060466d24ccca3/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic_fail.json
[modify] https://crrev.com/db9e62be87a2d0b2e8380bddf5060466d24ccca3/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic.json
[modify] https://crrev.com/db9e62be87a2d0b2e8380bddf5060466d24ccca3/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic_fail.json
[modify] https://crrev.com/db9e62be87a2d0b2e8380bddf5060466d24ccca3/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic.json
[modify] https://crrev.com/db9e62be87a2d0b2e8380bddf5060466d24ccca3/scripts/slave/recipes/swarming/deterministic_build.py

Comment 12 by thakis@chromium.org, Nov 20

Cc: dpranke@chromium.org mar...@chromium.org
Hm, that didn't work out -- turns out the .isolated files themselves are ironically not build-directory independent.

accessibility_unittests.isolated                                                                                                                                 : DIFFERENT (unexpected): 
  - files:
    - out\Release\VkICD_mock_icd.dll.pdb:  {u'h': u'9eb10e1b4fe15cbcdf3019324bec1b89a62578f7', u's': 12070912, u'm': 438} != None
    - out\Release.2\VkICD_mock_icd.dll.pdb:  None != {u'h': u'9eb10e1b4fe15cbcdf3019324bec1b89a62578f7', u's': 12070912, u'm': 438}
  - relative_cwd:  out\Release != out\Release.2



I suppose we could replace the build dir with some fixed string at isolate generation time? i.e. s:out/whatever:out/fixedbuilddir: ...somewhere (swarming code itself? mb?)

Comment 13 by thakis@chromium.org, Nov 20 

(Here's the one build with the new layout: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Windows%20deterministic/10767 See the compare_build_artifacts output.)

Comment 14 by dpranke@chromium.org, Nov 20 

Unlesss I'm missing something, it's important that the relative_cwd be what it is, because the swarming code is actually using that to figure out where it needs to launch the command from.

So, it might be that the entire premise of this bug is wrong, and you can't use two different build dirs (unless you also use two checkouts).

Comment 15 by thakis@chromium.org, Nov 21 

But the relative_cwd could just be a fixed string that's always the same, no? i.e. no matter what your build dir is, the isolated always contains out/fixedpath?

Comment 16 by thakis@chromium.org, Nov 21

Blockedon: 907488
Project Member

Comment 17 by bugdroid1@chromium.org, Nov 21 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/dda84a41a0fde29418aef182419a471ba6c3c7b6

commit dda84a41a0fde29418aef182419a471ba6c3c7b6
Author: Nico Weber <thakis@chromium.org>
Date: Wed Nov 21 14:59:15 2018

compare_build_artifacts.py: Add workaround for .isolated files containing the name of the build dir.

Bug: 899438,907488
Change-Id: Ic958d09e1fd4fc8a34f4f33fc9bfb9e3bc5d6a54
Reviewed-on: https://chromium-review.googlesource.com/c/1346670
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Hans Wennborg <hans@chromium.org>
Cr-Commit-Position: refs/heads/master@{#610051}
[modify] https://crrev.com/dda84a41a0fde29418aef182419a471ba6c3c7b6/tools/determinism/compare_build_artifacts.py
Project Member

Comment 18 by bugdroid1@chromium.org, Nov 21 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/4ec6ea74cd60b75f8a0c22d28600ebef612de917

commit 4ec6ea74cd60b75f8a0c22d28600ebef612de917
Author: Nico Weber <thakis@chromium.org>
Date: Wed Nov 21 15:07:32 2018

Revert "Move "Windows deterministic" bot back to old build dir layout for now."

This reverts commit db9e62be87a2d0b2e8380bddf5060466d24ccca3.

Reason for revert:
Now that https://chromium-review.googlesource.com/c/1346670 has landed,
this might work a bit better.

Original change's description:
> Move "Windows deterministic" bot back to old build dir layout for now.
> 
> The .isolated files contain the name of the build dir, so havingb
> build dirs out/Release and out/Release.2 makes the isolated comparison fail.
> Need to come up with a plan for that before relanding.
> 
> TBR=hans@chromium.org
> 
> Bug: 899438
> Change-Id: I058fd0db13eafe3f42307efd48d31db462261e6d
> Reviewed-on: https://chromium-review.googlesource.com/c/1342960
> Reviewed-by: Nico Weber <thakis@chromium.org>
> Commit-Queue: Nico Weber <thakis@chromium.org>

TBR=thakis@chromium.org,hans@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug: 899438
Change-Id: Ia03b896e0c8d989f8c9cb70080e692934deb0fea
Reviewed-on: https://chromium-review.googlesource.com/c/1346749
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>

[modify] https://crrev.com/4ec6ea74cd60b75f8a0c22d28600ebef612de917/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic_fail.json
[modify] https://crrev.com/4ec6ea74cd60b75f8a0c22d28600ebef612de917/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic.json
[modify] https://crrev.com/4ec6ea74cd60b75f8a0c22d28600ebef612de917/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic_fail.json
[modify] https://crrev.com/4ec6ea74cd60b75f8a0c22d28600ebef612de917/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic.json
[modify] https://crrev.com/4ec6ea74cd60b75f8a0c22d28600ebef612de917/scripts/slave/recipes/swarming/deterministic_build.py
Project Member

Comment 19 by bugdroid1@chromium.org, Nov 21 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b95b041b37185422ea0513a35d5ca22339faa94e

commit b95b041b37185422ea0513a35d5ca22339faa94e
Author: Nico Weber <thakis@chromium.org>
Date: Wed Nov 21 19:46:34 2018

Add debug logging to debug why https://chromium-review.googlesource.com/c/1346670 has no effect on the bot.

TBR=hans@chromium.org

Bug: 899438
Change-Id: Ibec766f0ccf5554556c4c578625469d20504fff9
Reviewed-on: https://chromium-review.googlesource.com/c/1347211
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#610178}
[modify] https://crrev.com/b95b041b37185422ea0513a35d5ca22339faa94e/tools/determinism/compare_build_artifacts.py

Comment 20 by thakis@chromium.org, Nov 21 

I filed issue 899438 for the isolated stuff. For now, I'll attempt to hack around it in compare_build_artifacts.py.
Project Member

Comment 21 by bugdroid1@chromium.org, Nov 22 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a08023e6014fe5be424ae87d9403e50dd9fa5422

commit a08023e6014fe5be424ae87d9403e50dd9fa5422
Author: Nico Weber <thakis@chromium.org>
Date: Thu Nov 22 02:41:01 2018

compare_build_artifacts.py: json-encode backslash in string before replacing

The string is used to replace things in a json-encoded string, so it
needs to be json-encoded.

TBR=hans

Bug: 899438
Change-Id: Iababe77ed7f8c2dafd0d0443f28f480082bc72d0
Reviewed-on: https://chromium-review.googlesource.com/c/1347312
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#610284}
[modify] https://crrev.com/a08023e6014fe5be424ae87d9403e50dd9fa5422/tools/determinism/compare_build_artifacts.py
Project Member

Comment 22 by bugdroid1@chromium.org, Nov 22 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/00e5353103a6dca9622e819e48138152fded70fc

commit 00e5353103a6dca9622e819e48138152fded70fc
Author: Nico Weber <thakis@chromium.org>
Date: Thu Nov 22 14:33:14 2018

Deterministic bots: Stop running api.isolate.isolate_tests().

Now that we're on gn, mb writes the .isolate file which lists all files in it,
and isolate_tests() then takes that .isolate file to create a .isolated file
containing hashes of all binaries, and names of hidden .tar archives that
have content-addressed names with all the contents of these files.

Creating the .isolated files is pretty slow (all build outputs are be
zipped up), the .isolated files contain the name of the build dir which
means it must be filtered out, and since the tar file names are content-addressed
those would have to be filtered out at compare time too, and the .isolated
files don't buy us anything we don't already get from the .isolate files
(compare_build_artifacts.py gets its file list from the .isolate files, not
from the .isolated files).

So stop running this step. It will make the bots cycle faster, and until
.isolated files are builddir-name independent it allows removing some
workarounds from compare_build_artifacts.py.

Bug: 907488,899438, 876915 
Change-Id: I416d5966755a1d199842bcdd94b80aed156b5594
Reviewed-on: https://chromium-review.googlesource.com/c/1347346
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Hans Wennborg <hans@chromium.org>

[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic_fail.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic__dbg_.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux_fail.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg__fail.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_Clang_deterministic_fail.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg_.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic_fail.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic__dbg__fail.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Mac_deterministic.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic_fail.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Windows_deterministic.json
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.py
[modify] https://crrev.com/00e5353103a6dca9622e819e48138152fded70fc/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic.json
Project Member

Comment 23 by bugdroid1@chromium.org, Nov 22 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/daa3cb99a97f9e52cce7d550e77f48ed519088c2

commit daa3cb99a97f9e52cce7d550e77f48ed519088c2
Author: Nico Weber <thakis@chromium.org>
Date: Thu Nov 22 18:52:02 2018

Update deterministic build whitelist now that "Windows deterministic" uses separate build dirs.

Also remove no-longer-needed debug logging.

TBR=hans

Bug: 899438
Change-Id: I1759e7e2f6ee8ece3fcdddeeb286754c1cf6b60a
Reviewed-on: https://chromium-review.googlesource.com/c/1348310
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#610467}
[modify] https://crrev.com/daa3cb99a97f9e52cce7d550e77f48ed519088c2/tools/determinism/compare_build_artifacts.py
[modify] https://crrev.com/daa3cb99a97f9e52cce7d550e77f48ed519088c2/tools/determinism/deterministic_build_whitelist.pyl
Project Member

Comment 24 by bugdroid1@chromium.org, Nov 26 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ed098fb72c9407e916bd4ca2b8875b1b675820d3

commit ed098fb72c9407e916bd4ca2b8875b1b675820d3
Author: Nico Weber <thakis@chromium.org>
Date: Mon Nov 26 15:13:01 2018

Update deterministm whitelist for "Windows deterministic".

The browser_console_module.js is apparently no longer needed.

Add

   'nacl_irt_x86_32.nexe',
   'nacl_irt_x86_32.nexe.debug',

which started misbehaving over the weekend.

Bug: 899438
Change-Id: I14a69aa7ff43203cc0dae4aef2db566db5d6cc12
Reviewed-on: https://chromium-review.googlesource.com/c/1351089
Reviewed-by: Hans Wennborg <hans@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#610847}
[modify] https://crrev.com/ed098fb72c9407e916bd4ca2b8875b1b675820d3/tools/determinism/deterministic_build_whitelist.pyl
Project Member

Comment 25 by bugdroid1@chromium.org, Nov 26 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/afcfd7a7700facd02ec30e0653ecfb2e83bfe601

commit afcfd7a7700facd02ec30e0653ecfb2e83bfe601
Author: Nico Weber <thakis@chromium.org>
Date: Mon Nov 26 15:14:16 2018

Switch "Deterministic Linux" (and (dbg) variant) to use to distinct build directories.

One of the two build directories now does incremental builds, so after
this the bots will check that build outputs are independent of the build path,
and that incremental and full builds produce the same outputs.

Bug: 899438
Change-Id: I5f1a32b4cf61e1b877efa0caf693a1d9544d613a
Reviewed-on: https://chromium-review.googlesource.com/c/1351109
Reviewed-by: Hans Wennborg <hans@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>

[modify] https://crrev.com/afcfd7a7700facd02ec30e0653ecfb2e83bfe601/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic.json
[modify] https://crrev.com/afcfd7a7700facd02ec30e0653ecfb2e83bfe601/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux_fail.json
[modify] https://crrev.com/afcfd7a7700facd02ec30e0653ecfb2e83bfe601/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux.json
[modify] https://crrev.com/afcfd7a7700facd02ec30e0653ecfb2e83bfe601/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg__fail.json
[modify] https://crrev.com/afcfd7a7700facd02ec30e0653ecfb2e83bfe601/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_Deterministic_Linux__dbg_.json
[modify] https://crrev.com/afcfd7a7700facd02ec30e0653ecfb2e83bfe601/scripts/slave/recipes/swarming/deterministic_build.expected/full_chromium_swarm_linux_chromium_clobber_deterministic_fail.json
[modify] https://crrev.com/afcfd7a7700facd02ec30e0653ecfb2e83bfe601/scripts/slave/recipes/swarming/deterministic_build.py

Comment 26 by thakis@chromium.org, Nov 26

Blockedon: 908463
Project Member

Comment 27 by bugdroid1@chromium.org, Nov 26 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1858e05748cf85040fb29f4ddb748854eeef7faa

commit 1858e05748cf85040fb29f4ddb748854eeef7faa
Author: Nico Weber <thakis@chromium.org>
Date: Mon Nov 26 16:43:46 2018

Update "Deterministic Linux" build whitelist after moving to distinct build dirs.

Also remove a few entries that haven't been used since we made the bots
use filelists from .isolate files for comparison.

TBR=hans

Bug: 899438
Change-Id: Iba9e3d111063c352552a5af51cc14ba3d8219a29
Reviewed-on: https://chromium-review.googlesource.com/c/1351370
Reviewed-by: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#610857}
[modify] https://crrev.com/1858e05748cf85040fb29f4ddb748854eeef7faa/tools/determinism/deterministic_build_whitelist.pyl

Comment 28 by thakis@chromium.org, Nov 26

Blockedon: 908474
Project Member

Comment 29 by bugdroid1@chromium.org, Nov 26 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/14a495f33d87c7465f94e07d3edc444f87b253c9

commit 14a495f33d87c7465f94e07d3edc444f87b253c9
Author: Nico Weber <thakis@chromium.org>
Date: Mon Nov 26 17:28:08 2018

Update "Deterministic Linux (dbg)" build whitelist after moving to distinct build dirs.

TBR=hans

Bug: 899438
Change-Id: I790c63caa60a56fa5102c46ac10ff57aa22e535b
Reviewed-on: https://chromium-review.googlesource.com/c/1351375
Reviewed-by: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#610869}
[modify] https://crrev.com/14a495f33d87c7465f94e07d3edc444f87b253c9/tools/determinism/deterministic_build_whitelist.pyl
Project Member

Comment 30 by bugdroid1@chromium.org, Nov 26 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bb6843b1270f1692f29e948bd57ab2e5b7c9a747

commit bb6843b1270f1692f29e948bd57ab2e5b7c9a747
Author: Nico Weber <thakis@chromium.org>
Date: Mon Nov 26 19:11:38 2018

Stop listing .isolated files in deterministic build whitelist.

Unused after https://chromium-review.googlesource.com/c/chromium/tools/build/+/1347346

Also remove code that's been unused since that change related to .isolated
files from compare_build_artifacts.py

Bug: 899438, 876915 
Change-Id: Ie572c2a27924b72a2ffb86165afca37fcbd667d6
Reviewed-on: https://chromium-review.googlesource.com/c/1351378
Reviewed-by: Erik Chen <erikchen@chromium.org>
Commit-Queue: Nico Weber <thakis@chromium.org>
Cr-Commit-Position: refs/heads/master@{#610906}
[modify] https://crrev.com/bb6843b1270f1692f29e948bd57ab2e5b7c9a747/tools/determinism/compare_build_artifacts.py
[modify] https://crrev.com/bb6843b1270f1692f29e948bd57ab2e5b7c9a747/tools/determinism/deterministic_build_whitelist.pyl

Comment 31 by dpranke@chromium.org, Nov 26 

re #c15:
> But the relative_cwd could just be a fixed string that's always the same, no?
> i.e. no matter what your build dir is, the isolated always contains out/fixedpath?

Assuming I understand your question correctly, no. The relative_cwd is actually used by swarming when the task is launched, and it needs to be right, i.e., it needs to point to the actual build directory.

I don't think you can "fix" this in the sense of making two isolates built in two different build directories be the same without essentially implementing the work in bug 907585.

Comment 32 by thakis@chromium.org, Nov 27 

dpranke: mind saying that on issue 907488? Then I'll reply there.
Project Member

Comment 33 by bugdroid1@chromium.org, Nov 27 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/eb5fc54df5fc7343b059a7afec023c50715cb688

commit eb5fc54df5fc7343b059a7afec023c50715cb688
Author: Nico Weber <thakis@chromium.org>
Date: Tue Nov 27 02:39:04 2018

Disable symbols for nacl-compiler-built object files.

The non-pnacl gcc-based nacl compilers embed absolute paths in their debug info.
This is bad for goma usage and build determinism.  Since I've never
seen a stack in nacl-built code and neither the code nor the toolchain
are changing much anymore, just disable symbols for nacl-built code.
(The same code when built with a non-gcc nacl toolchain of course still
has symbols, including the nacl-clang-built IRT where we archive the symbols for
production builds.)

Bug: 899438,906741,429358
Change-Id: I3062a26f25281759cc240f1967ab670f28e562a6
Reviewed-on: https://chromium-review.googlesource.com/c/1351492
Reviewed-by: Takuto Ikuta <tikuta@chromium.org>
Reviewed-by: Derek Schuff <dschuff@chromium.org>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#610998}
[modify] https://crrev.com/eb5fc54df5fc7343b059a7afec023c50715cb688/build/config/compiler/BUILD.gn

Comment 34 by dpranke@chromium.org, Nov 27 

> dpranke: mind saying that on issue 907488? Then I'll reply there.

Done. I knew I saw another bug on this topic go by ...
Project Member

Comment 35 by bugdroid1@chromium.org, Nov 27 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/11e2b3ba0e2876e256b0ce48e5d4a34a92eff985

commit 11e2b3ba0e2876e256b0ce48e5d4a34a92eff985
Author: Nico Weber <thakis@chromium.org>
Date: Tue Nov 27 10:26:52 2018

Remove a few whitelist entries for the "Windows deterministic" and "Deterministic Linux (dbg)" bots.

These are no longer nondeterministic following
https://chromium-review.googlesource.com/c/chromium/src/+/1351492

Bug: 899438
Change-Id: I2f72531111fdc62c340032df610fcd256e524504
Reviewed-on: https://chromium-review.googlesource.com/c/1351721
Commit-Queue: Hans Wennborg <hans@chromium.org>
Reviewed-by: Hans Wennborg <hans@chromium.org>
Cr-Commit-Position: refs/heads/master@{#611062}
[modify] https://crrev.com/11e2b3ba0e2876e256b0ce48e5d4a34a92eff985/tools/determinism/deterministic_build_whitelist.pyl

Comment 36 by thakis@chromium.org, Nov 30

Blocking: 910699

Comment 37 by thakis@chromium.org, Dec 3

Blocking: 910465

Comment 38 by thakis@chromium.org, Dec 21

Blockedon: 917355

Comment 39 by thakis@chromium.org, Jan 3

Blockedon: 918882

Comment 40 by thakis@chromium.org, Jan 7

Blockedon: 919437

Comment 41 by thakis@chromium.org, Jan 7

Blockedon: 912946

Comment 42 by benhenry@google.com, Jan 11

Status: Assigned (was: Untriaged)
This issue has an owner, a component and a priority, but is still listed as untriaged or unconfirmed. By definition, this bug is triaged. Changing status to "assigned". Please reach out to me if you disagree with how I've done this.
►

Sign in to add a comment