Katie, mind looking into adding this? If you don’t have time just let me know.

Nope, I can look into it. 

These builds were stuck on bot_update, which we know to be slow at times. See crbug.com/635641. I'm going to discuss with tandrii why we didn't just add a timeout to the bot_update step. It also probably makes sense to add an overall build timeout for this master. It looks like successful builds take between five minutes and four and a half hours. This is what I found with manual inspection. I wrote a query that returned no results, so I'm going to dig into that too. A six hour timeout seems reasonable. It's long for many of the builders, but at least would avoid builds hanging for >100 hours, as they did in  crbug.com/676200 .

Thanks katthomas@ for looking into this.

Six hours does sound reasonable. Note there are some non-Swarmed testers on this waterfall which do take a pretty long time to run tests since the test suites are large. If it's possible to configure shorter timeouts for different machines then maybe we could institute that for the builders on this waterfall (distinct from the testers). A good set to start with would be these:

https://build.chromium.org/p/chromium.gpu.fyi/builders/GPU%20Win%20Builder
https://build.chromium.org/p/chromium.gpu.fyi/builders/GPU%20Win%20Builder%20%28dbg%29
https://build.chromium.org/p/chromium.gpu.fyi/builders/GPU%20Win%20x64%20Builder
https://build.chromium.org/p/chromium.gpu.fyi/builders/GPU%20Win%20x64%20Builder%20%28dbg%29
https://build.chromium.org/p/chromium.gpu.fyi/builders/GPU%20Win%20Clang%20Builder%20%28dbg%29
https://build.chromium.org/p/chromium.gpu.fyi/builders/GPU%20Mac%20Builder
https://build.chromium.org/p/chromium.gpu.fyi/builders/GPU%20Mac%20Builder%20%28dbg%29
https://build.chromium.org/p/chromium.gpu.fyi/builders/GPU%20Linux%20Builder
https://build.chromium.org/p/chromium.gpu.fyi/builders/GPU%20Linux%20Builder%20%28dbg%29
https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20ChromiumOS%20Builder
https://build.chromium.org/p/chromium.gpu.fyi/builders/Linux%20ChromiumOS%20Ozone%20Builder


Note that https://build.chromium.org/p/chromium.gpu.fyi/builders/GPU%20Mac%20Builder is stuck again. I just filed Issue 677373 about it.

It's certainly possible. Here's a CL for that. 

https://chromium-review.googlesource.com/424126

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/e5b16ced2a2e2eb8e0e73ff52b002c5c81f638cb

commit e5b16ced2a2e2eb8e0e73ff52b002c5c81f638cb
Author: Katie Thomas <katthomas@google.com>
Date: Tue Jan 03 17:13:01 2017

Add timeout to chromium.gpu.fyi builders

BUG= 676343 

Change-Id: I87400c3266ea7dd6d94b5436415d15df54fce164
Reviewed-on: https://chromium-review.googlesource.com/424334
Commit-Queue: Katie Thomas <katthomas@google.com>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/e5b16ced2a2e2eb8e0e73ff52b002c5c81f638cb/masters/master.chromium.gpu.fyi/master.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/8b8c0663ca2dc16688841a39a72f30c318737d7a

commit 8b8c0663ca2dc16688841a39a72f30c318737d7a
Author: Katie Thomas <katthomas@google.com>
Date: Thu Jan 05 01:42:47 2017

Add per builder timeouts

BUG= 676343 

Change-Id: Id639746e6199fe1ffe4fa5218ba1a69993454831
Reviewed-on: https://chromium-review.googlesource.com/424126
Reviewed-by: Paweł Hajdan Jr. <phajdan.jr@chromium.org>

[modify] https://crrev.com/8b8c0663ca2dc16688841a39a72f30c318737d7a/masters/master.chromium.gpu.fyi/slaves.cfg
[modify] https://crrev.com/8b8c0663ca2dc16688841a39a72f30c318737d7a/scripts/master/recipe_master_helper.py

Thanks for adding this!