Also for Fuchsia ARM64 Cast Audio:
https://build.chromium.org/p/chromium.linux/builders/Fuchsia%20ARM64%20Cast%20Audio

With both Exception steps failed compile, e.g.
https://build.chromium.org/p/chromium.linux/builders/Fuchsia%20ARM64%20Cast%20Audio/builds/226

And Exception steps failed package build, e.g.
https://build.chromium.org/p/chromium.linux/builders/Fuchsia%20ARM64%20Cast%20Audio/builds/225

x64 has been in bad health for a long time, some green but mostly purple.
ARM64 has been almost entirely purple for a long time, with a few green exceptions.

The functional error can be see in the "stdio" log:

command timed out: 2400 seconds without output, attempting to kill
process killed by signal 9

BuildBot will timeout and kill the build after a set amount of time has passed without any new I/O. It looks like this is currently set to 40 minutes.

It's not clear why it's timing out, but the last log line is:
[5779/5787] STAMP obj/content/test/content_unittests__test_runner_script.stamp
Loading streams...

Either consider extending the timeout or figure out what's part of the build is taking 40 minutes (historically, this tends to be linking).

Indeed, it looks like the linker is the hold-up. Hopped on the bot when it was stuck and grabbed the process tree for ninja:

ninja-linux64(21134)─┬─sh(2815)───python(2816)───clang++(2818)───ld.lld(2819)─┬─{ld.lld}(2872)
                     │                                                        ├─{ld.lld}(2873)
                     │                                                        ├─{ld.lld}(2874)
                     │                                                        ├─{ld.lld}(2875)
                     │                                                        ├─{ld.lld}(2876)
                     │                                                        ├─{ld.lld}(2877)
                     │                                                        ├─{ld.lld}(2878)
                     │                                                        └─{ld.lld}(2879)
                     ├─sh(2846)───python(2848)───clang++(2852)───ld.lld(2854)─┬─{ld.lld}(2880)
                     │                                                        ├─{ld.lld}(2881)
                     │                                                        ├─{ld.lld}(2882)
                     │                                                        ├─{ld.lld}(2883)
                     │                                                        ├─{ld.lld}(2884)
                     │                                                        ├─{ld.lld}(2885)
                     │                                                        ├─{ld.lld}(2886)
                     │                                                        └─{ld.lld}(2887)
                     └─sh(2850)───python(2851)───clang++(2853)───ld.lld(2855)─┬─{ld.lld}(2888)
                                                                              ├─{ld.lld}(2889)
                                                                              ├─{ld.lld}(2890)
                                                                              ├─{ld.lld}(2891)
                                                                              ├─{ld.lld}(2892)
                                                                              ├─{ld.lld}(2893)
                                                                              ├─{ld.lld}(2894)
                                                                              └─{ld.lld}(2895)

And it was frozen at that until the build timedout. I doubt it's WAI that linking takes that long when building only 2 targets (cast_shell and cast_test_lists). I respawned the machine in the case it'll help, but I don't have my hopes up. 

Going to drop the symbol level on both of the cast bots. I think they're unintentionally building at symbol_level=2 thanks to https://codesearch.chromium.org/chromium/src/build/config/compiler/compiler.gni?rcl=7f7beb7dc8691f39a273aaa3f0915fabe3ceee50&l=197, while their non-cast-audio counterparts build w/ symbol_level=0.

Thanks, yes, that's the problem alright. wez@ and I debugged the same thing a while back, I thought one of us had a CL in progress, sorry for leaving it broken. :(

Re #6: Yeah, I was about to post a CL for that, didn't realise jbudorick@ was looking at it.

Re #5: In principle we should be able to build these binaries even with symbol_level=2 - a cleaner fix might be to limit link-step concurrency. Note that it's not building "two targets"; there are at least three substantial binaries that get built.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2515aaa6ba76a733add94c363a3fcb45cbdbe701

commit 2515aaa6ba76a733add94c363a3fcb45cbdbe701
Author: John Budorick <jbudorick@chromium.org>
Date: Wed Nov 29 20:28:18 2017

Drop the symbol_level on fuchsia cast audio bots to 0.

Bug:  789534 
Change-Id: Ice59510e006bcb8b8a70bbac862d52f0e6a3a40e
Reviewed-on: https://chromium-review.googlesource.com/797213
Reviewed-by: Scott Graham <scottmg@chromium.org>
Commit-Queue: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#520226}
[modify] https://crrev.com/2515aaa6ba76a733add94c363a3fcb45cbdbe701/tools/mb/mb_config.pyl

Looks fixed now!

compile is, but package build is still hosed on the ARM64 Cast Audio bot :/