Full log snippet:
[4/8511] ACTION //base:base_build_config_gen__apply_gcc(//build/toolchain/android:android_clang_arm64)
FAILED: gen/base/base_build_config_gen/java_cpp_template/org/chromium/base/BuildConfig.java 
python ../../build/android/gyp/gcc_preprocess.py --depfile gen/base/base_build_config_gen_BuildConfig.d --include-path ../../ --output gen/base/base_build_config_gen/java_cpp_template/org/chromium/base/BuildConfig.java --template=../../base/android/java/templates/BuildConfig.template --defines _DCHECK_IS_ON
Traceback (most recent call last):
  File "../../build/android/gyp/gcc_preprocess.py", line 56, in <module>
    sys.exit(main(sys.argv[1:]))
  File "../../build/android/gyp/gcc_preprocess.py", line 46, in main
    DoGcc(options)
  File "../../build/android/gyp/gcc_preprocess.py", line 29, in DoGcc
    build_utils.CheckOutput(gcc_cmd)
  File "/b/c/b/android_arm64_rel/src/build/android/gyp/util/build_utils.py", line 169, in CheckOutput
    stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=cwd, env=env)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory


Example failing builds:
https://build.chromium.org/p/tryserver.webrtc/builders/android_arm64_rel/builds/21102
https://build.chromium.org/p/tryserver.webrtc/builders/android_more_configs/builds/8363

It only hits our trybots, so I'm not sure why it's happening.

chrome-bot@slave721-c4:(Linux 14.04):~$ which gcc; echo $?
1

Your trybots don't have gcc in their PATH, it seems.

Are they supposed to? I thought we used the same image as everyone else. 
Adding labs.

Oddly, that only appears to be the case for some (but not all) of the slaves in that group?

I hope labs can help us out here and maybe do a puppet sync/update or something like that.

Is it right we depend on gcc at all though, I thought it was deprecated from the NDK and not even used for building Chromium anymore?

This is manual intervention at play.  Probably someone trying to get binutils-aarch64-linux-gnu installed on these webrtc slaves (see crbug.com/747829 for context)

chrome-bot@slave721-c4:(Linux 14.04):~$ history
    1  2017/08/10 08:00:24 PDT  cd /b/c/git_cache/
    2  2017/08/10 08:00:26 PDT  ll
    3  2017/08/10 08:00:30 PDT  rm -Rf *
    4  2017/08/17 04:18:50 PDT  ls
    5  2017/08/17 04:18:55 PDT  cd
    6  2017/08/17 04:18:56 PDT  ls
    7  2017/08/17 04:19:23 PDT  cd /b/c/b/linux_rel/src/build
    8  2017/08/17 04:19:29 PDT  ./install-build-deps.sh

From slave721-c4:/var/log/apt/term.log:

Log started: 2017-08-17  04:20:43
(Reading database ... ^M(Reading database ... 5%^M(Reading database ... 10%^M(Reading database ... 15%^M(Reading database ... 20%^M(Reading database ... 25%^M(Reading database ... 30%^M(Reading database ... 35%^M(Reading database ... 40%^M(Reading database ... 45%^M(Reading database ... 50%^M(Reading database ... 55%^M(Reading database ... 60%^M(Reading database ... 65%^M(Reading database ... 70%^M(Reading database ... 75%^M(Reading database ... 80%^M(Reading database ... 85%^M(Reading database ... 90%^M(Reading database ... 95%^M(Reading database ... 100%^M(Reading database ... 189619 files and directories currently installed.)^M
Removing build-essential (11.6ubuntu6) ...^M
Removing xutils-dev (1:7.7~1ubuntu6) ...^M
Removing libtool (2.4.2-1.7ubuntu1) ...^M
Removing g++ (4:4.8.2-1ubuntu6) ...^M
Removing gcc (4:4.8.2-1ubuntu6) ...^M
Removing libgnome2-0:amd64 (2.32.1-4ubuntu1) ...^M
Removing libgnome2-bin (2.32.1-4ubuntu1) ...^M
Removing libbonobo2-0:amd64 (2.32.1-0ubuntu5) ...^M
Removing liborbit2:amd64 (1:2.14.19-0.3) ...^M
Removing libidl0:amd64 (0.8.14-0.2ubuntu4) ...^M
Removing libidl-common (0.8.14-0.2ubuntu4) ...^M
Removing cpp (4:4.8.2-1ubuntu6) ...^M

I'm guessing update-alternates did not play nice here and /usr/bin/gcc was removed as 4.8.4-2ubuntu1~14.04.3 is actually installed.

I suggest that these affected gce instances be re-spawned?

#5: this is for running the preprocessor. It needs a host binary. The NDK isn't involved.

Re #6: yes it was probably me installing binutils-aarch64-linux-gnu on the machines to green up our builds after reshuffling slaves to more VMs did Android arm64 builds. Did that remove GCC?
Please do whatever is best to recover these. I assume it will be installed on respawn as it's now a part of install-build-deps.sh? 
For similar cases in the future, is respawn always preferred or?

gce images with the latest install-build-deps.sh are now being built thanks to crbug.com/741237   However an older image is still being used in production.  That will be fixed soon. 

In the meantime I think slave721-c4 is the only bad actor which I have manually fixed. 

I looked around and couldn't find any more failures similar to this. So let's close this for now.

Thanks for identifying the problem and fixing it!