New issue
Advanced search Search tips
Starred by 2 users

Issue metadata

Status: Fixed
Closed: Dec 7
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug

issue 869348
issue 912946

Sign in to add a comment

Issue 911761: Deterministic bots red after deleteIcon.png was deleted

Reported by, Dec 4 Project Member

Issue description


Different list of files in both directories:

One build dir does incremental builds, the other clobber builds.

However, the bot does run

'C:\b\swarming\w\ir\cipd_bin_packages\python.exe' -u 'C:\b\swarming\w\ir\kitchen-checkout\recipe_engine\recipe_modules\file\resources\' --json-output 'c:\b\swarming\w\ir\tmp\t\tmp5jwxes.json' rmglob 'C:\b\swarming\w\ir\cache\builder\src\out\Release' '*.isolate'

("rm old .isolate") to delete the .isolate files in the incremental build dir.

We then regen the isolate files by running

'C:\b\swarming\w\ir\cipd_bin_packages\python.exe' -u 'C:\b\swarming\w\ir\cache\builder\src\tools\mb\' isolate-everything -m -b 'Windows deterministic' --config-file 'C:\b\swarming\w\ir\cache\builder\src\tools\mb\mb_config.pyl' --goma-dir 'C:\b\swarming\w\ir\cache\goma\client' //out/Release

generate .isolate files

Comment 2 by, Dec 4

Fix idea: Get file list from clobber build dir, and then when getting file list from incremental build dir and we walkdir() dirs in the isolate files there, use the results from the clobber dir walks.

This helps the bot. It doesn't help the trybots, which all use incremental build dirs and which because of that zip up nondeterministic isolated files that they send to swarming.

Comment 3 by, Dec 4

This is the file that copies all the resources to out/resources/inspector:

(If the data dep was from a checked-in file, then git would've removed said file)

Comment 4 by, Dec 5

 Issue 912204  has been merged into this issue.

Comment 5 by, Dec 6

Here's a list of all the places where we put generated dirs in data_deps:*%5C%22%5C$%5B%5E%22%5D*%5C/

It's not that many places. We should probably just forbid having dirs below generated dirs in data_deps.

Comment 7 by, Dec 7

Status: Started (was: Assigned)

Comment 8 by, Dec 7

Project Member
The following revision refers to this bug:

commit f213164d219da210c4c0383d4878c02633599965
Author: Nico Weber <>
Date: Fri Dec 07 00:20:39 2018

Explicitly list all $root_out_dir/resources/inspector files in the isolate.

Listing a generated directory in the isolate means that incremental
builders (and devs doing incremental builds) can have stale files in
their generated isolated files.

Bug:  911761 
Change-Id: I8652b272b44b031771e4458d43d5091abb093428
Commit-Queue: Nico Weber <>
Reviewed-by: Dmitry Gozman <>
Cr-Commit-Position: refs/heads/master@{#614502}

Comment 9 by, Dec 7

Blocking: 912946

Comment 10 by, Dec 7

Blocking: 869348
Issue 912946 for the follow-up of doing this for other generated dirs.

Comment 11 by, Dec 7

Status: Fixed (was: Started)

Sign in to add a comment