failure on my own build: https://chromium-swarm.appspot.com/task?id=3d640d84947e3210

passing tests, from the tree:

Builder:

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win%20x64%20Builder%20%28dbg%29/1018

rev: b6370aecafa241206aa031d26344effbb2d8b50f

goma_dir = "C:\\b\\swarming\\w\\ir\\cache\\goma_client"
is_component_build = true
is_debug = true
strip_absolute_paths_from_debug_symbols = true
symbol_level = 1
use_goma = true

Tester:

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win10%20Tests%20x64%20%28dbg%29/468

Results:

https://chromium-swarm.appspot.com/task?id=3d646a56bc592e10 -> PASS

This makes little sense, it's almost like I have a broken build environment.

reducing this down - these two isolated inputs seem to be the same (they are based on the same files) but one works and one does not.

33cd707b87bf27d1277467e6c6b44de1dbf208a5 -> https://chromium-swarm.appspot.com/user/task/3d786f88f59de210 -> works
f55ee67904f46e1bcf84018821d86d22f9c300ed -> https://chromium-swarm.appspot.com/user/task/3d786c215979d010 -> does not work.

hmm these archives are different:

working: https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=33cd707b87bf27d1277467e6c6b44de1dbf208a5

not-working: https://isolateserver.appspot.com/browse?namespace=default-gzip&digest=f55ee67904f46e1bcf84018821d86d22f9c300ed

I'm confused, the 'working' one has a tar file and the 'not working' has all the UCRT DLLs packaged. Am I packaging the isolate files incorrectly? +maruel

right, thanks to maruel we worked out what was happening here. This is because when manually isolating the files, the read_only bit is set to 1, meaning the isolated inputs are read-only on the bots. This causes one of the tests to fail.

The reason why this was not failing on the bots is that the isolated inputs from the bots do not have the read_only bit set - this is a regression when moving to tarred isolate files.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/2dab862746c25ffc557be831d63268ebe5d99995

commit 2dab862746c25ffc557be831d63268ebe5d99995
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Mon May 14 18:31:26 2018

[client] better handle tarfile created on other OSes

When a tarfile is created, the paths are with native path separator. Sometimes
we may want to fetch the input on another OS (e.g. Windows files on a linux
host) for inspection, and this would create files with a '\' in their name
instead of a path separator.

Bug:  841878 
Change-Id: I1352ae1fde59fcbbe40720375c9f340721239156
Reviewed-on: https://chromium-review.googlesource.com/1057424
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Quinten Yearsley <qyearsley@chromium.org>

[modify] https://crrev.com/2dab862746c25ffc557be831d63268ebe5d99995/client/isolateserver.py

The rest of the work in the isolate code is going to be tracked as issue 842802.

This will be fixed in https://chromium-review.googlesource.com/c/chromium/src/+/1058327

As commented on the CL, I'm not sure it's entirely satisfactory to have a different behavior on developer's machines vs. the trybots here.

We'd ideally like to be able to detect these failures earlier than 'running on swarming'. Would there be any way of making a change to the build environment to mark output files read-only without breaking the build? Adding bruce as the build-guru to have a think about this and comment...

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/02c5fa7600185dc28e3e5a9c8823caa23beccb41

commit 02c5fa7600185dc28e3e5a9c8823caa23beccb41
Author: Will Harris <wfh@chromium.org>
Date: Mon May 14 23:59:03 2018

Fix sandbox dynamic code tests to work with read-only test binaries.

BUG= 841878 

Change-Id: I70db1aed7130e124cb5e257ac31e4b5711e78963
Reviewed-on: https://chromium-review.googlesource.com/1058327
Reviewed-by: Penny MacNeil <pennymac@chromium.org>
Commit-Queue: Will Harris <wfh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#558537}
[modify] https://crrev.com/02c5fa7600185dc28e3e5a9c8823caa23beccb41/sandbox/win/src/process_mitigations_dyncode_unittest.cc

It's no different from the fact that running a test in a non-isolated checkout has access to all files, not only the ones specified in data and data_deps. This is a much bigger deal than the +w bit in term of frequency of breakage.

Client side tooling to run isolated tests exists but is not polished, that's out of scope of this specific issue.

okay, how about we meet half way and make "isolate.py remap" mark the files read-only - then at least it would be caught there.

my commands:

$ python tools\swarming_client\isolate.py remap -i out\Debug_x64\sbox_integration_tests.isolate -o d:\src\isolated\sbox_integration_tests

$ attrib d:\src\isolated\sbox_integration_tests\out\Debug_x64\sbox_integration_tests.exe
A                    d:\src\isolated\sbox_integration_tests\out\Debug_x64\sbox_integration_tests.exe

"isolate.py remap" should make the files read-only once my CL to add it into mb.py gets committed. If not, it's a bug.

okay great - is that happening on issue 842802?

Yep!