we have improved the toolchain builders quite a bit but it is very cumbersome to go through all of them to find why they failed.

There is a new tool described at:

https://docs.google.com/document/d/1KvX2Aj1rAYGAUxBDjq6IicmyH694KoaoblBzCrI0fJY/edit#

(it also contains the location of the tool) 

That generates reports like this:

----------------------------------
Waterfall from 2016-09-09 07:47:19
Report created 2016-09-09 08:25:46
81 green, 12 red canaries (12.9% failure rate)
----------------------------------
Failure counts:
10 CanaryCompletion
    celes/376: 3/41
    nyan/381: 3/41 (last 2)
    expresso/541: 3/41 (last 2)
    kunimitsu/376: 2/41
    kip/455: 3/41 (last 2)
    x86-alex/5196: 2/41
    orco/377
    auron/378: 4/41
    gnawty/376: 3/41
    auron_yuna/376: 4/41
2 HWTest [bvt-inline]
    daisy/377 (DUTs): 3/41 (last 3)
    kevin/384 (swarming timeout)
1 AUTest [au]
    daisy/377 (DUTs): 3/41 (last 3)


I think we should be able to modify this tool to do it for toolchain builders only.

Assigning to next sheriff (Rahul).