Labs folks, could you please take a look at this machine? It's vm49-m1 . Thanks.

Clobbered the slave.  Effective starting with https://build.chromium.org/p/chromium.gpu.fyi/builders/Mac%20GPU%20ASAN%20Release/builds/3081

Still failing with the "Unable to walk" message.

Thanks Peter for trying. It looks to me like either:

 - the directory out/Release/test_data is being created, symlinked, hardlinked, etc. during the build process and it's unreadable

or

 - one of the isolates is referring to it, and it's not being put in place during the build process

Code search isn't helpful. Could the pixel wrangler log on to the bot and try to figure out which of these it is?

Could Swarming's batcharchive operation produce a more helpful error message in this case?

Well the directory is not created. When you do a clean build locally with the target as specified in this build, is this directory present?

according to
https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.gpu.fyi%2FMac_GPU_ASAN_Release%2F2920%2F%2B%2Frecipes%2Fsteps%2Fcompile%2F0%2Fstdout
the list is:
angle_end2end_tests angle_unittests audio_unittests browser_tests gl_tests gl_unittests gles2_conform_test services_unittests swiftshader_unittests telemetry_gpu_integration_test

Next pixel wrangler will need to try doing an ASAN build locally and see what's going on.

I've built those targets on the bot, and the directory was not created.
I've also checked out the sources at the revision of the successful build
https://luci-milo.appspot.com/buildbot/chromium.gpu.fyi/Mac%20GPU%20ASAN%20Release/2919, and test_data was not created either. 

I've tried building on my Linux workstation. With the same flags as the bot, test_data is not created. If I remove "is_asan = true", test_data is created.

Seems this is due to https://codereview.chromium.org/2865723002 disabling JS tests on ASAN. Looks to me like the bug is that isolate is trying to look for these tests, while it shouldn't. I'm not familiar with isolates, Ken, can you please look further?

Yuly, thanks for tracking that down. It's surprising that the fix for  Issue 579666  broke our ASAN bot but I'll track down what's going on.

https://chromium-review.googlesource.com/560560/ up for review fixing it.

For the record, the way I tracked this down was to build the targets from c#6 above with is_asan=true enable_nacl=false (which is what the bot uses), then:

./tools/mb/mb.py isolate out/Release angle_end2end_tests
./tools/mb/mb.py isolate out/Release ...

then grepped for test_data in out/Release/*.isolate .

Realizing it was a runtime dependency of browser_tests and confirming with:
gn desc out/Release //chrome/test:browser_tests runtime_deps

found the spot in the GN files where it was being unconditionally added.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fcdd8da1bcce96e08c7b301ed94368417ec09c34

commit fcdd8da1bcce96e08c7b301ed94368417ec09c34
Author: Kenneth Russell <kbr@chromium.org>
Date: Thu Jul 06 03:35:23 2017

Fix browser_tests' runtime dependency on test_data.

If the sanitizers are turned on, or NaCl is turned off, then the
top-level test_data directory in the output directory won't be
created. Conditionalize it as a runtime dependency.

BUG= 732815 

Change-Id: I9d3c54ae85ae02792f4ddc8d071ba23d84cc09bb
Reviewed-on: https://chromium-review.googlesource.com/560560
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#484455}
[modify] https://crrev.com/fcdd8da1bcce96e08c7b301ed94368417ec09c34/chrome/test/BUILD.gn

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/adb6ffe9685ba87bd998582cd06600da17b69a27

commit adb6ffe9685ba87bd998582cd06600da17b69a27
Author: Elly Fong-Jones <ellyjones@chromium.org>
Date: Thu Jul 06 14:26:47 2017

Revert "Fix browser_tests' runtime dependency on test_data."

This reverts commit fcdd8da1bcce96e08c7b301ed94368417ec09c34.

Reason for revert:

MSAN Mac javascript_browser_tests are now failing like:
../../chrome/test/base/javascript_browser_test.cc:54: Failure
Value of: PathService::Get(chrome::DIR_GEN_TEST_DATA, &gen_test_data_directory)

Original change's description:
> Fix browser_tests' runtime dependency on test_data.
> 
> If the sanitizers are turned on, or NaCl is turned off, then the
> top-level test_data directory in the output directory won't be
> created. Conditionalize it as a runtime dependency.
> 
> BUG= 732815 
> 
> Change-Id: I9d3c54ae85ae02792f4ddc8d071ba23d84cc09bb
> Reviewed-on: https://chromium-review.googlesource.com/560560
> Reviewed-by: Dirk Pranke <dpranke@chromium.org>
> Commit-Queue: Kenneth Russell <kbr@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#484455}

TBR=phajdan.jr@chromium.org,dpranke@chromium.org,kbr@chromium.org,jochen@chromium.org

Change-Id: I47f3ea9807eb4aab575f30041cf66460e505d332
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  732815 
Reviewed-on: https://chromium-review.googlesource.com/561816
Reviewed-by: Elly Fong-Jones <ellyjones@chromium.org>
Commit-Queue: Elly Fong-Jones <ellyjones@chromium.org>
Cr-Commit-Position: refs/heads/master@{#484579}
[modify] https://crrev.com/adb6ffe9685ba87bd998582cd06600da17b69a27/chrome/test/BUILD.gn

The essential difference between these two Mac ASAN builders:
https://build.chromium.org/p/chromium.memory/builders/Mac%20ASan%2064%20Builder?numbuilds=200
https://build.chromium.org/p/chromium.gpu.fyi/builders/Mac%20GPU%20ASAN%20Release

is that the chromium.memory bot builds both the browser_tests and unit_tests targets. The chromium.gpu.fyi bot builds only browser_tests.

src/chrome/test/base/javascript_browser_test.cc asserts that PathService::Get(chrome::DIR_GEN_TEST_DATA, ...) succeeds, so it expects that (in this case) out/Release/test_data will exist. See the failures in e.g.:
https://luci-milo.appspot.com/buildbot/chromium.memory/Mac%20ASan%2064%20Tests%20%281%29/32124

From downloading the isolate for browser_tests from a successful build like:
https://build.chromium.org/p/chromium.memory/builders/Mac%20ASan%2064%20Tests%20%281%29/builds/32125
with the hash:
5a01edee77fa7bed9297d941723ede1ace2acf3b

it is clear that the *only* reason that out/Release/test_data exists is to provide the directory out/Release/test_data/chrome/safe_browsing_dmg/ . This data is generated by the unit_tests target, not browser_tests. The only reason the chromium.memory Mac ASAN bot is working is that it builds both browser_tests and unit_tests.

In a non-ASAN build, a lot of WebUI test resources are generated into out/Release/test_data/chrome/browser/ and out/Release/test_data/chrome/test/ .

I think the fixes needed are:
 - Be more precise, and don't unilaterally include the generated test_data directory as a runtime_dep of browser_tests. Only include the appropriate subdirectories.
 - Change src/chrome/test/base/javascript_browser_test.cc so it doesn't assert that it can find the generated test_data directory. It's only needed for a subset of the webui tests.

Proceeding with these changes.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b856e626264db71e88787d8e2a778c891ff053f0

commit b856e626264db71e88787d8e2a778c891ff053f0
Author: Kenneth Russell <kbr@chromium.org>
Date: Mon Jul 10 19:38:17 2017

Fix overly-broad runtime dependency on generated test_data dir.

browser_tests depends on $root_out_dir/test_data, which is not
guaranteed to be generated by that target. Other dependencies already
capture the needed subdirectories in that top-level directory.

WebUI browser_tests currently assert that that directory exists. As of
the fix for  Issue 579666 , it does if the sanitizers (ASAN, MSAN,
TSAN) are disabled, or if NaCl is enabled.

The only reason the build is succeeding on the ASAN bot right now is
that that directory is being created on behalf of the unit_tests
target, which is compiled simultaneously on that machine.

Fix the runtime dependency, and conditionalize the assertion in the
test harness.

BUG= 732815 

Change-Id: Ia6d64af3bb3eb88aaa45188c353ed08c5b1434e3
Reviewed-on: https://chromium-review.googlesource.com/563040
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Reviewed-by: Jochen Eisinger <jochen@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#485349}
[modify] https://crrev.com/b856e626264db71e88787d8e2a778c891ff053f0/chrome/test/BUILD.gn
[modify] https://crrev.com/b856e626264db71e88787d8e2a778c891ff053f0/chrome/test/base/javascript_browser_test.cc

Just got this compiling and hadn't gotten around to getting the bot fully green when GN started failing to run on the bot. Filed Issue 742004. This particular issue is fixed though.