https://codereview.chromium.org/2078293002 seems very suspicious.

... because it didn't modify checklicenses: https://codesearch.chromium.org/chromium/src/tools/checklicenses/checklicenses.py?rcl=0&l=650

I wonder why this isn't showing up on the main waterfall bot https://build.chromium.org/p/chromium.linux/builders/Android%20Tests/builds/28746 ...?

Don't know. Revert just landed, though.

#3: theory: this failed on gce hosts that previously ran an N5X (i.e., arm64) build since LARNG shares a trigger GCE pool with android_n5x_swarming_rel?

Ah, that could very well be the case. Those two bots definitely shouldn't be sharing slave-dirs. It's pretty much a clobber each time they switch builds.

Hmm, good point, I hadn't thought of that.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build.git/+/f79668abb95a1d762e1d3da6bb04113d7ecec5b7

commit f79668abb95a1d762e1d3da6bb04113d7ecec5b7
Author: bpastene <bpastene@chromium.org>
Date: Fri Jul 08 23:07:22 2016

Split arm and arm64 android trybots work directories.

Also delete android_swarming_rel trybot.

BUG= 626784 

Review-Url: https://codereview.chromium.org/2137733002

[modify] https://crrev.com/f79668abb95a1d762e1d3da6bb04113d7ecec5b7/masters/master.tryserver.chromium.android/builders.pyl
[modify] https://crrev.com/f79668abb95a1d762e1d3da6bb04113d7ecec5b7/scripts/slave/recipe_modules/chromium_tests/trybots.py
[delete] https://crrev.com/cf5cdaa14775c27153ec44868e2a33513789198f/scripts/slave/recipes/chromium_trybot.expected/full_tryserver_chromium_android_android_swarming_rel.json

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build_limited/scripts/slave/+/a7018f6952c46c69d74741cf73420658c14e3a1a

commit a7018f6952c46c69d74741cf73420658c14e3a1a
Author: recipe-roller <recipe-roller@chromium.org>
Date: Fri Jul 08 23:18:22 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/740d62aef55b787b739cdd4668c9ed1da8b15f0b

commit 740d62aef55b787b739cdd4668c9ed1da8b15f0b
Author: recipe-roller <recipe-roller@chromium.org>
Date: Fri Jul 08 23:28:37 2016

Roll recipe dependencies (trivial).

This is an automated CL created by the recipe roller. This CL rolls recipe
changes from upstream projects (e.g. depot_tools) into downstream projects
(e.g. tools/build).

More info is at https://goo.gl/zkKdpD. Use https://goo.gl/noib3a to file a bug
(or complain)

build:
  https://crrev.com/f79668abb95a1d762e1d3da6bb04113d7ecec5b7 Split arm and arm64 android trybots work directories. (bpastene@chromium.org)

R=bpastene@chromium.org
BUG= 626784 

TBR=martiniss@chromium.org,phajdan.jr@chromium.org

Review-Url: https://codereview.chromium.org/2133863004

[modify] https://crrev.com/740d62aef55b787b739cdd4668c9ed1da8b15f0b/infra/config/recipes.cfg

This is still happening frequently.
https://build.chromium.org/p/tryserver.chromium.android/builders/linux_android_rel_ng?numbuilds=200

I guess I shouldn't be surprised given that /build/linux/debian_*-sysroot/ shows up in .gitignore. We may need to clobber some hosts.

(host checkouts, that is, not just output directories)

Yeah, since these are gce, we can do a mass respawn that will wipe all local checkouts. The problem is doing this with as little disruption as possible. I think I might just do it all in one go later tonight. Since these failures aren't actively blocking anything, I think it can wait that long. Feel free to tell me otherwise if you disagree.

That sounds OK.

I respawned them all yesterday evening. We should be good.