... I'm guessing this is because all of the Linux bots are quarantined? https://chromium-swarm.appspot.com/botlist?c=id&c=os&c=task&c=status&f=pool%3AChrome.LUCI&l=100&q=pool%3A&s=id%3Aasc

updating title to reflect that this is ongoing

[+migration experts]

Thanks for reporting this. Nodir, Dan, any idea how we can fix this and prevent it in the future?

+Ben mentioned that it's related to the robolectric/junit tests that LUCI linux_android_rel_ng was trying to run.

I am on it

there is a ton of android-tmp-roboelectric-* dirs in /tmp

bpastene@ said it happens in JUnit +mikecase@ 

bots are fixed
https://chromium-swarm.appspot.com/botlist?c=id&c=os&c=task&c=status&f=pool%3AChrome.LUCI&f=os%3ALinux&l=100&q=os&s=id%3Aasc

and builds has startedhttps://ci.chromium.org/buildbucket/luci.chromium.try/LUCI%20linux_chromium_rel_ng

but we still need to fix the cause. Repurposing this bug and assigning to mikecase

Bots are dead again. Can we disable LUCI LARNG until this is resolved?

Why? Is this actually causing a problem? These builders do not impact production.

they block John and Dirk from landing CLs to src. I'll send a CL.

Nevermind, that's just lcrng. Is it because it's using the swarming resources?

AFAICT, LUCI linux_android_rel_ng runs on the same segment of the Chrome.LUCI pool as LUCI linux_chromium_rel_ng. Having LUCI LARNG blacklist all of the machines used by LUCI LCRNG indeed blocks Dirk and I from landing CLs to src via the CQ.

e.g. https://chromium-review.googlesource.com/c/583288/#message-3c7238882cf04f3063a6a6b6d449edf8aaa5e06e

This looks like a bug? CQ shouldn't be failing builds because of an experimental builder.

Also could we actually fix Android tests to not leave thousands of artifacts?

Wow, able to really repro this locally. Sooooooooooo many temp files left behind.

A solution could be overriding System.setProperty("java.io.tmpdir") at the beginning of our JunitTestMain class and then nuke that directory at the end of the test.

Will code that up.

Our builders export a tempdir that you can totally use via the "TMPDIR" environment variable. It looks like Java doesn't use this (it really should...), so if you just do:

-Djava.io.tmpdir=$TMPDIR

That will work fine. If you use this, you won't have to clean up anything, as the builder will automatically purge it at end of build.

Oooh, that is nice.

I imagine this might also affect developers on their local machines if they run lots of Robolectric tests. Might just do the cleanup in the Junit tset runner for their sake.

Because of the load-splitting we're doing, they aren't actually experimental for jbudorick@ and myself; those bots are counted *instead* of the buildbot equivalents.

That said, I'm not likely to be landing something in src for the next day or two, so it doesn't matter as much to me (I can always NOTRY if needed, too).

I see, sorry, estaab@ explained that to me offline.

RE #16: mikecase@, if you wouldn't mind, could you do the quick property fix for the Android stuff just so we can move past this bot-disabling problem? I'd imagine something like (it's late so in random pseudocode):

if defined ENVIRONMENT['TMPDIR'] {
  opts += '-Djava.io.tmpdir=' + ENVIRONMENT['TMPDIR']
}

This should make it so that TMPDIR is overridden in Java the same way that it is on the system at all times.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8c5e159ad4cb7a954a6266f87ac03d08584a0765

commit 8c5e159ad4cb7a954a6266f87ac03d08584a0765
Author: Mike Case <mikecase@google.com>
Date: Tue Jul 25 00:18:26 2017

Roll Robolectric to version that has fix for cleaning up tmp files.

BUG= 747204 

Change-Id: If8f93ac272bd02ec7a7b5f44ada2c5e4adc0cf45
Reviewed-on: https://chromium-review.googlesource.com/583901
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Michael Case <mikecase@chromium.org>
Cr-Commit-Position: refs/heads/master@{#489155}
[modify] https://crrev.com/8c5e159ad4cb7a954a6266f87ac03d08584a0765/DEPS
[modify] https://crrev.com/8c5e159ad4cb7a954a6266f87ac03d08584a0765/third_party/robolectric/README.chromium

Pretty sure Robolectric creating temp files and not cleaning them up is now fixed (at least fix worked locally for me).

Is there anything left for this bug?

the bots in Chrome.LUCI are still all quarantined.

I respawned all of the afflicted instances, so they are no longer quarantined. If the problem is truly solved, they should remain that way :D

Thanks Dan. Fingers crossed!