Troopers, can you help? Is this broken everywhere or just our bots?

I don't see any difference in the gclient revision, tools/build revision, VM, results or timing of previous steps. The only thing that's different is the timing of the package_build step, and it looks like network is slower? (more lines of "uploading"...).

If you look at the last 200 builds for these builders, the failures due to package_build are all clustered together. It looks like the step is timing out. Perhaps transient network errors? I'm closing for now. Please reopen if it becomes an issue again. 

Thanks for investigating kattthomas@. Yes, let's reopen if it happens again -- it does sound like the labs' network connection got saturated.

Happening again today:
https://build.chromium.org/p/chromium.gpu/builders/GPU%20Win%20Builder/builds/67985
https://build.chromium.org/p/chromium.gpu/builders/GPU%20Win%20Builder%20%28dbg%29/builds/59594

It's happening around the same time of the day. Not sure if it's because of the load or other factors...

Issue 750856 has been merged into this issue.

Looks like the builders are non functional in the most busy 8 hours of the day.
This severely impacts our ability to triage regressions.
Can we please get more network capacity?

Also happening for last few builds on GPU Win Builder and GPU Win Builder (Dbg):

https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.gpu%2FGPU_Win_Builder%2F68087%2F%2B%2Frecipes%2Fsteps%2Fpackage_build%2F0%2Fstdout

https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.gpu%2FGPU_Win_Builder__dbg_%2F59677%2F%2B%2Frecipes%2Fsteps%2Fpackage_build%2F0%2Fstdout

Logdog is failing to load complete logs in both cases and the last step is uploading the isolate using gsutil.

ccing people who might have more context on this step

These builds were killed. You can see this by clicking the build-level "stdio" at the top of the build:

command timed out: 3600 seconds elapsed, attempting to kill


LogDog will not have terminal logs because the logging agent gets killed along with the build. The problem is that the build exceeded 3600 seconds without emitting any output.  My guess is that the GS upload stalled.

Windows bots timing out when uploading to GS? Sounds an awful lot like bug 748261. Could be the same root cause.