New issue
Advanced search Search tips
Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Dec 7
Cc:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug

Blocking:
issue 869348
issue 912946



Sign in to add a comment
link

Issue 911761: Deterministic bots red after deleteIcon.png was deleted

Reported by thakis@chromium.org, Dec 4 Project Member

Issue description

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Windows%20deterministic/10950

containing

https://chromium-review.googlesource.com/c/chromium/src/+/1357602


Different list of files in both directories:
  resources\inspector\Images\deleteIcon.png



One build dir does incremental builds, the other clobber builds.

However, the bot does run

'C:\b\swarming\w\ir\cipd_bin_packages\python.exe' -u 'C:\b\swarming\w\ir\kitchen-checkout\recipe_engine\recipe_modules\file\resources\fileutil.py' --json-output 'c:\b\swarming\w\ir\tmp\t\tmp5jwxes.json' rmglob 'C:\b\swarming\w\ir\cache\builder\src\out\Release' '*.isolate'

https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8928114177155101424/+/steps/rm_old_.isolate/0/stdout

("rm old .isolate") to delete the .isolate files in the incremental build dir.


We then regen the isolate files by running

'C:\b\swarming\w\ir\cipd_bin_packages\python.exe' -u 'C:\b\swarming\w\ir\cache\builder\src\tools\mb\mb.py' isolate-everything -m chromium.win -b 'Windows deterministic' --config-file 'C:\b\swarming\w\ir\cache\builder\src\tools\mb\mb_config.pyl' --goma-dir 'C:\b\swarming\w\ir\cache\goma\client' //out/Release


generate .isolate files
 

Comment 2 by thakis@chromium.org, Dec 4

Fix idea: Get file list from clobber build dir, and then when getting file list from incremental build dir and we walkdir() dirs in the isolate files there, use the results from the clobber dir walks.

This helps the bot. It doesn't help the trybots, which all use incremental build dirs and which because of that zip up nondeterministic isolated files that they send to swarming.

Comment 3 by thakis@chromium.org, Dec 4

This is the file that copies all the resources to out/resources/inspector: https://cs.chromium.org/chromium/src/third_party/blink/renderer/devtools/BUILD.gn?type=cs&q=resources%5C/inspector+file:%5C.gn&sq=package:chromium&g=0&l=986

(If the data dep was from a checked-in file, then git would've removed said file)

Comment 4 by thakis@chromium.org, Dec 5

Cc: thakis@chromium.org mar...@chromium.org dpranke@chromium.org rogerm@chromium.org
 Issue 912204  has been merged into this issue.

Comment 5 by thakis@chromium.org, Dec 6

Here's a list of all the places where we put generated dirs in data_deps: https://cs.chromium.org/search/?q=%5Cbdata%5B%5E%5D%5D*%5C%22%5C$%5B%5E%22%5D*%5C/%5C%22+file:%5C.gn+pcre:yes&sq=package:chromium&type=cs

It's not that many places. We should probably just forbid having dirs below generated dirs in data_deps.

Comment 7 by thakis@chromium.org, Dec 7

Status: Started (was: Assigned)

Comment 8 by bugdroid1@chromium.org, Dec 7

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f213164d219da210c4c0383d4878c02633599965

commit f213164d219da210c4c0383d4878c02633599965
Author: Nico Weber <thakis@chromium.org>
Date: Fri Dec 07 00:20:39 2018

Explicitly list all $root_out_dir/resources/inspector files in the isolate.

Listing a generated directory in the isolate means that incremental
builders (and devs doing incremental builds) can have stale files in
their generated isolated files.

Bug:  911761 
Change-Id: I8652b272b44b031771e4458d43d5091abb093428
Reviewed-on: https://chromium-review.googlesource.com/c/1366443
Commit-Queue: Nico Weber <thakis@chromium.org>
Reviewed-by: Dmitry Gozman <dgozman@chromium.org>
Cr-Commit-Position: refs/heads/master@{#614502}
[modify] https://crrev.com/f213164d219da210c4c0383d4878c02633599965/BUILD.gn
[modify] https://crrev.com/f213164d219da210c4c0383d4878c02633599965/third_party/blink/renderer/devtools/BUILD.gn

Comment 9 by thakis@chromium.org, Dec 7

Blocking: 912946

Comment 10 by thakis@chromium.org, Dec 7

Blocking: 869348
Issue 912946 for the follow-up of doing this for other generated dirs.

Comment 11 by thakis@chromium.org, Dec 7

Status: Fixed (was: Started)

Sign in to add a comment