Don't cc troopers, mark Infra-Troopers label instead.
This is particularly so for EMEA troopers which don't handle Pri1 issues.

+mcgreevy@ +maruel@ because isolate.

Oh, and I suspect this actually due to us exhausting our network pipe in where your bots are. +Labs to confirm this and possibly dedup.

And here is an issue 753184 about downlink saturating. Since that issue is internal, I keep this public bug blocked on that internal one.

Actually, I am not sure my first analysis was correct. The hanging has been happening even during low MTV activity. So, there are probably two bugs here:

1. isolate shouldn't hang if file isn't found. I thought that was fixed, but apparently not.

2. if some file is indeed missing but should be there, there is some misconfiguration in the builder/gn/some other config.


Log extract from [1] which strongly suggests network isn't an issue here:

06:29:16.432578 PushDirectory(E:\b\c\b\win\src\third_party\webgl\) = 15469 files
06:29:16.432578 PushDirectory(E:\b\c\b\win\src\tools\perf\chrome_telemetry_build\) = 4 files
[I2017-08-08T06:29:16.445579-07:00 4520 0 archiver.go:617] Looked up 50 items
06:29:16.463581 PushDirectory(E:\b\c\b\win\src\tools\perf\core\) = 22 files
[I2017-08-08T06:29:16.525587-07:00 4520 0 archiver.go:617] Looked up 50 items
telemetry_gpu_integration_test  GetFileAttributesEx E:\b\c\b\win\src\out\Debug_x64\blink_web.dll.pdb: The system cannot find the file specified.
gl_tests  GetFileAttributesEx E:\b\c\b\win\src\out\Debug_x64\blink_web.dll.pdb: The system cannot find the file specified.
[I2017-08-08T06:29:16.553590-07:00 4520 0 archiver.go:617] Looked up 50 items
angle_white_box_tests  GetFileAttributesEx E:\b\c\b\win\src\out\Debug_x64\blink_web.dll.pdb: The system cannot find the file specified.
[I2017-08-08T06:29:16.817616-07:00 4520 0 archiver.go:617] Looked up 50 items
[I2017-08-08T06:29:16.835618-07:00 4520 0 archiver.go:617] Looked up 50 items
[I2017-08-08T06:29:16.895624-07:00 4520 0 archiver.go:617] Looked up 50 items
[30764] [6266/1.20Gib/29482] [6132/6266] [11/223.0Mib/160/623.8Mib] 15s
[I2017-08-08T06:29:16.988634-07:00 4520 0 archiver.go:617] Looked up 50 items
[I2017-08-08T06:29:17.030638-07:00 4520 0 archiver.go:637] Uploaded 17.8Mib: out\Debug_x64\media.dll.pdb
[I2017-08-08T06:29:17.158651-07:00 4520 0 archiver.go:617] Looked up 50 items
[I2017-08-08T06:29:17.454680-07:00 4520 0 archiver.go:637] Uploaded 16.7Mib: out\Debug_x64\angle_end2end_tests.exe.pdb
[30764] [6268/1.33Gib/29482] [6232/6268] [13/257.6Mib/160/623.8Mib] 16s
[I2017-08-08T06:29:18.114746-07:00 4520 0 archiver.go:637] Uploaded 16.3Mib: out\Debug_x64\base.dll.pdb
[30764] [6269/1.73Gib/29482] [6232/6269] [14/273.8Mib/160/623.8Mib] 17s
[I2017-08-08T06:29:18.924827-07:00 4520 0 archiver.go:637] Uploaded 19.6Mib: out\Debug_x64\libGLESv2.dll
[I2017-08-08T06:29:19.443879-07:00 4520 0 archiver.go:637] Uploaded 13.9Mib: out\Debug_x64\angle_end2end_tests.exe
[I2017-08-08T06:29:19.625897-07:00 4520 0 archiver.go:637] Uploaded 37.7Mib: out\Debug_x64\angle_unittests.exe.pdb
[I2017-08-08T06:29:19.638899-07:00 4520 0 archiver.go:637] Uploaded 16.7Mib: out\Debug_x64\media.dll
[I2017-08-08T06:29:19.690904-07:00 4520 0 archiver.go:637] Uploaded 16.4Mib: out\Debug_x64\skia.dll.pdb
[30764] [6269/1.73Gib/29482] [6232/6269] [19/378.0Mib/160/623.8Mib] 18s
# Terminated ~1 hour later by buildbot due to no stdout output.

[1] https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Ftryserver.chromium.angle%2Fwin_angle_x64_dbg_ng%2F5200%2F%2B%2Frecipes%2Fsteps%2Fisolate_tests%2F0%2Fstdout

I think your analysis is correct, given the breakage outside of MTV hours.  It is probably an isolate client bug in that case.  Maruel I think you're the de-facto owner of the isolate client? Can you look or reassign?

Any updates here?  This blocks ANGLE from testing or landing any CLs.

See also similar/duplicate  issue 753774 .