This looks like this CL, landed yesterday: https://codereview.chromium.org/2163833003/

+stip@, +nicholaslin@, who own this CL.

Did anybody start working on that? If no, I will revert this CL. The bot is down for many hours. 

It is impossible to revert it. Please fix it ASAP.

Note: ATM this bug is marked P3. If you really need ASAP, please elevate the priority to something more appropriate.

 Issue 637339  has been merged into this issue.

This test was added yesterday, and the patch didn't pick up this new test.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/aa986de5b33958a7f4fbf27c5d1a8a5c50ac4c0f

commit aa986de5b33958a7f4fbf27c5d1a8a5c50ac4c0f
Author: nicholaslin <nicholaslin@google.com>
Date: Fri Aug 12 20:03:50 2016

Update chromium.android.json

BUG= 637213 
NOTRY=True
TBR=jbudorick@chromium.org

Review-Url: https://codereview.chromium.org/2240923002
Cr-Commit-Position: refs/heads/master@{#411749}

[modify] https://crrev.com/aa986de5b33958a7f4fbf27c5d1a8a5c50ac4c0f/build/android/test_wrapper/logdog_wrapper.py
[modify] https://crrev.com/aa986de5b33958a7f4fbf27c5d1a8a5c50ac4c0f/testing/buildbot/chromium.android.json
[modify] https://crrev.com/aa986de5b33958a7f4fbf27c5d1a8a5c50ac4c0f/testing/buildbot/chromium.linux.json

The change seems to be fixing the mojo_test_apk failures. The cause of the failure was due to mojo_test_apk not having logdog cipd package specification in the testing json. An additional check to skip calling logdog in the case where it's not installed (i.e. new test, local testing) has been added to logdog_wrapper.

Builds are currently failing with:
E  107.166s Main  Failed to locate logcat for device 06be45b2003b9b9f
Traceback (most recent call last):
  File "/b/swarm_slave/w/irPmdMY1/build/android/pylib/local/device/local_device_environment.py", line 192, in TearDown
    with open(m.output_file, 'r') as infile:
IOError: [Errno 2] No such file or directory: '/b/swarm_slave/w/it94Qg9Y/tmp9Z6kFp/06be45b2003b9b9f_20160812T210836'
E  108.084s Main  Unrecognized error occurred.
Traceback (most recent call last):
  File "/b/swarm_slave/w/irPmdMY1/build/android/test_runner.py", line 1004, in main
    return RunTestsCommand(args)
  File "/b/swarm_slave/w/irPmdMY1/build/android/test_runner.py", line 822, in RunTestsCommand
    return RunTestsInPlatformMode(args)
  File "/b/swarm_slave/w/irPmdMY1/build/android/test_runner.py", line 943, in RunTestsInPlatformMode
    all_raw_results, args.json_results_file)
  File "/b/swarm_slave/w/irPmdMY1/build/android/pylib/base/environment.py", line 33, in __exit__
    self.TearDown()
  File "/b/swarm_slave/w/irPmdMY1/build/android/pylib/local/device/local_device_environment.py", line 207, in TearDown
    [m.output_file for m in self._logcat_monitors])
  File "/b/swarm_slave/w/irPmdMY1/third_party/catapult/devil/devil/utils/file_utils.py", line 29, in MergeFiles
    raise e
IOError: [Errno 2] No such file or directory: '/b/swarm_slave/w/it94Qg9Y/tmp9Z6kFp/06be45b2003b9b9f_20160812T210836'


Is this related to the new CL?

I don't think so. Could you add the build link?

This looks a device died/could not be connected to during the middle of a test.

This also affects event_unittests: https://uberchromegw.corp.google.com/i/chromium.linux/builders/Android%20Tests/builds/30448

Yep, unrelated to logdog. 06be45b2003b9b9f is bad, taking that bot (build3-b4) offline.

E   90.133s prepare_device(06be45b2003b9b9f)  Shard timed out: prepare_device(06be45b2003b9b9f)
Traceback (most recent call last):
  File "/b/swarm_slave/w/irAml4U9/build/android/pylib/local/device/local_device_environment.py", line 55, in wrapper
    return f(dev, *args, **kwargs)
  File "/b/swarm_slave/w/irAml4U9/build/android/pylib/local/device/local_device_environment.py", line 138, in prepare_device
    monitor.Start()
  File "/b/swarm_slave/w/irAml4U9/third_party/catapult/devil/devil/android/logcat_monitor.py", line 191, in Start
    self._adb.Logcat(clear=True)
  File "/b/swarm_slave/w/irAml4U9/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 589, in Logcat
    return self._RunDeviceAdbCmd(cmd, timeout, retries).splitlines()
  File "/b/swarm_slave/w/irAml4U9/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 278, in _RunDeviceAdbCmd
    check_error=check_error)
  File "/b/swarm_slave/w/irAml4U9/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/swarm_slave/w/irAml4U9/third_party/catapult/devil/devil/utils/timeout_retry.py", line 157, in Run
    error_log_func=error_log_func)
  File "/b/swarm_slave/w/irAml4U9/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 186, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/swarm_slave/w/irAml4U9/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 158, in _JoinAll
    thread.ReraiseIfException()
  File "/b/swarm_slave/w/irAml4U9/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 81, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/swarm_slave/w/irAml4U9/third_party/catapult/devil/devil/utils/timeout_retry.py", line 150, in <lambda>
    child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
  File "/b/swarm_slave/w/irAml4U9/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarm_slave/w/irAml4U9/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 245, in _RunAdbCmd
    timeout_retry.CurrentTimeoutThreadGroup().GetRemainingTime())
  File "/b/swarm_slave/w/irAml4U9/third_party/catapult/devil/devil/utils/cmd_helper.py", line 273, in GetCmdStatusAndOutputWithTimeout
    raise TimeoutError(output.getvalue())

Ran `sudo service swarming stop` on build3-b4.labs.

kolos@, did you mean to remove the Sheriff-Chromium label? Removing it now, it sounds like its being worked on.

I believe we can close this, as the bot is green again.