Looks like this was happening intermittently but fairly regularly between about 10/12-22:22 UTC and 10/13-02:22 UTC, and hasn't happened since. I haven't been able to identify any specific cause from the logs. Resolving since this appears to have gone away.

Reactivating this issue as there have been a few more flakes in the last 12 or so hours.

Since it still appears to be a timeout issue, adding the infra-troopers tag.

I'm not seeing examples where it's a timeout? Could you add a link?

It's definitely possible that I'm not reading the logs correctly, but here is an example from  https://uberchromegw.corp.google.com/i/chromium.linux/builders/Android%20Tests/builds/33362/steps/content_unittests%20on%20Android/logs/stdio

E 1285.004s run_tests_on_device(06adc6ac003b6d51)  Shard timed out: run_tests_on_device(06adc6ac003b6d51)

I 1346.615s run_tests_on_device(0717f034003bca6a)  Still working on StartInstrumentation(0717f034003bca6a, org.chromium.native_test/org.chromium.native_test.NativeTestInstrumentationTestRunner, retries=0, extras={'org.chromium.native_test.NativeTestInstrumentationTestRunner.ShardNanoTimeout': 120000000000, 'org.chromium.native_test.NativeTestInstrumentationTestRunner.NativeTestActivity': 'org.chromium.native_test.NativeUnitTestActivity', 'org.chromium.native_test.NativeTestInstrumentationTestRunner.TestList': '/data/local/tmp/temp_file-6028d20f38efe'}, timeout=120.0, raw=False)
C 1406.592s run_tests_on_device(0717f034003bca6a)  Timed out. Dumping threads.

So far today I have seen this twice on the Android builder.

This looks like a timeout in catapult to me, maybe? Adding some folks in case they see something.

devil/pylib --> jbudorick, rnephew

The way the android test runner works is that it will break the tests up into chunks of 256, and attempt to run them. One of these chunks seems to be timing out.

E  185.066s run_tests_on_device(06b2c6d9003bfabf)  Shard timed out: run_tests_on_device(06b2c6d9003bfabf)
Traceback (most recent call last):
  File "/b/swarm_slave/w/irRQvqQ0/build/android/pylib/local/device/local_device_environment.py", line 55, in wrapper
    return f(dev, *args, **kwargs)
  File "/b/swarm_slave/w/irRQvqQ0/build/android/pylib/local/device/local_device_test_run.py", line 62, in run_tests_on_device
    result = self._RunTest(dev, test)
  File "/b/swarm_slave/w/irRQvqQ0/build/android/pylib/local/device/local_device_gtest_run.py", line 363, in _RunTest
    timeout=timeout, retries=0)
  File "/b/swarm_slave/w/irRQvqQ0/build/android/pylib/local/device/local_device_gtest_run.py", line 154, in Run
    self._component, extras=extras, raw=False, **kwargs)
  File "/b/swarm_slave/w/irRQvqQ0/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/swarm_slave/w/irRQvqQ0/third_party/catapult/devil/devil/utils/timeout_retry.py", line 159, in Run
    error_log_func=error_log_func)
  File "/b/swarm_slave/w/irRQvqQ0/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 186, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/swarm_slave/w/irRQvqQ0/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 151, in _JoinAll
    (len(alive_threads), len(self._threads)))
CommandTimeoutError: Timed out waiting for 1 of 1 threads.

This means one or more of the tests are taking longer than they once did or a test that takes a long time has been added.

We can do several things to try and fix this. Increase the timeout, find out what CL introduced it taking longer(probably the hardest one to do), or decrease the number of tests per shard.

John is out today, so adding mikecase@ as well.

The right thing to do here is the hard one: figure out what's taking so long and fix it. (and perhaps try to make it easier to identify such issues)

None of the last 200 runs have red content_unittests steps. Is this still an issue, and if it is is it still a p1 issue?

Removing Infra-Troopers since it hasn't happened since the 18th.

Detected 3 new flakes for test/step "content_unittests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKQsSBUZsYWtlIh5jb250ZW50X3VuaXR0ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

The latest failures are "shard #0 expired, not enough capacity"