Adding Troopers since I'm unable to figure out what's causing these problems. I've logged into a machine to investigate but some process is altering the /b/c mounts so it's hard to debug locally it seems.

With "other cases" I meant that most builds fail, but the one I clobbered went green. There are some other green builds too though, which is hard to explain.

Our main bots that are used in the CQ are:
https://build.chromium.org/p/tryserver.webrtc/builders/android_arm64_rel
https://build.chromium.org/p/tryserver.webrtc/builders/android_rel
https://build.chromium.org/p/tryserver.webrtc/builders/android_dbg
https://build.chromium.org/p/tryserver.webrtc/builders/android_compile_x64_dbg
https://build.chromium.org/p/tryserver.webrtc/builders/android_compile_x86_dbg
and they're all affected by this.

Interestingly, the Android Clang trybot always succeeds: 
https://build.chromium.org/p/tryserver.webrtc/builders/android_clang_dbg
which points at this has something to do with the Android default toolchain (GCC).

Hmm, could it be that clobbering is different when done from landmine than when triggered from the web UI?

These are a trybot builds with clobber triggered by my landmine (happens in the ruhooks step):
https://build.chromium.org/p/tryserver.webrtc/builders/android_compile_x86_rel/builds/4500 (which happens to be green)
https://uberchromegw.corp.google.com/i/tryserver.webrtc/builders/android_compile_x86_rel/builds/4501 (fails)

but this build was clobbered from the UI:
https://build.chromium.org/p/tryserver.webrtc/builders/android_compile_x64_dbg/builds/10038 (green)

I just triggered two new clobbers from the web UI to gather more data:
https://build.chromium.org/p/tryserver.webrtc/builders/android_compile_x64_dbg/builds/10051
https://build.chromium.org/p/tryserver.webrtc/builders/android_compile_x86_dbg/builds/9953

After looking into what https://cs.chromium.org/chromium/src/build/clobber.py does compared to the Web UI clobber (which wipes all of out/Debug or out/Release), the former seems to leave args.gn and writes a build.ninja file.

To be sure, I'm going "nuke it from orbit", so I'm cleaning out all the out/ dirs on these slaves using:
for s in slave719-c4 slave720-c4 slave721-c4 slave722-c4 slave723-c4 slave724-c4 slave725-c4 slave837-c4 slave838-c4 slave839-c4 slave840-c4 slave1210-c4 slave1211-c4 slave1212-c4 slave1213-c4 slave1214-c4 slave1215-c4 slave1216-c4 slave1217-c4 slave1218-c4 slave1219-c4; do ssh $s rm -rf /b/c/b/android_*_dbg/src/out; done;

Let's see if that greens up the next builds.

Oops, I'm doing another cleaning using rm -rf /b/c/b/android_*_rel/src/out to cover the remaining builders.

Unfortunately the cleaning didn't help.
I'm trying additional web UI clobbering at:
https://uberchromegw.corp.google.com/i/tryserver.webrtc/builders/android_rel/builds/19598
https://uberchromegw.corp.google.com/i/tryserver.webrtc/builders/android_dbg/builds/19760
https://uberchromegw.corp.google.com/i/tryserver.webrtc/builders/android_compile_x64_dbg/builds/10054
https://uberchromegw.corp.google.com/i/tryserver.webrtc/builders/android_compile_x86_dbg/builds/9956
https://uberchromegw.corp.google.com/i/tryserver.webrtc/builders/android_compile_x86_rel/builds/4511
https://uberchromegw.corp.google.com/i/tryserver.webrtc/builders/android_compile_arm64_rel/builds/461
https://uberchromegw.corp.google.com/i/tryserver.webrtc/builders/android_compile_arm64_dbg/builds/493

And a few non-clobbered builds without trybot patches:
https://uberchromegw.corp.google.com/i/tryserver.webrtc/builders/android_compile_x64_dbg/builds/10055
https://uberchromegw.corp.google.com/i/tryserver.webrtc/builders/android_compile_x86_dbg/builds/9957
https://uberchromegw.corp.google.com/i/tryserver.webrtc/builders/android_compile_x86_rel/builds/4512
https://uberchromegw.corp.google.com/i/tryserver.webrtc/builders/android_compile_arm64_rel/builds/462
https://uberchromegw.corp.google.com/i/tryserver.webrtc/builders/android_compile_arm64_dbg/builds/494

every one of those builds succeeded. Does this need to still be in trooper queue?

I fired them to be sure I wasn't just lucky in earlier attempts. The problem is that we have like 25 VMs that share a pool and powers ~10 builders. That means that even if I clobber many times, I cannot be sure each local checkout has picked up the clobbering.
Only a landmine could guarantee that, but apparently it doesn't clean up what's causing this (which I still don't know exactly what it may be).

Since this is a one-time thing, I'm now trying a more brutal approach: wipe all the android checkouts on these machines and let them be re-created the next build (Git cache should make this reasonably fast).
I'll put it back to the trooper queue in case that doesn't work.

+seanmccullough@chromium.org so you can see my reply in #9.

So, I wiped /b/c/b/android* on all these VMs and triggered new builds. All passed except the ones for android_rel and android_dbg:
https://uberchromegw.corp.google.com/i/tryserver.webrtc/builders/android_rel/builds/19607
https://uberchromegw.corp.google.com/i/tryserver.webrtc/builders/android_dbg/builds/19769
There's nothing special with those configs, so apparently that fix also doesn't work reliably. Could it be needed to wipe the Git cache as well? Maybe there's something weird cached for the third_party/android_tools or something (although that shouldn't really have changed after our restructuring).

+jbudorick who have detailed insight in this toolchain.

I had another look today and now all bots look healthy. I'm closing as fixed and will reopen if I see another failure. The autoroller bot will keep triggering builds during the holidays.