New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 606292 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Apr 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 1
Type: Bug

Blocking:
issue 605864
issue 606250



Sign in to add a comment

bisect can not run with an error, "Exception steps exception device_status_check"

Project Member Reported by toyoshim@chromium.org, Apr 25 2016

Issue description

Blocking: 605864
Blocking: 606250
Cc: pras...@chromium.org ligim...@chromium.org
Components: Tests>AutoBisect
Labels: BisectMonitor Infra OS-Android
The exception is observed continuously from Build #2925 to #2933.

Link to builder :
==================
https://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/2933

Link to Log:
============
https://build.chromium.org/p/tryserver.chromium.perf/builders/android_nexus7_perf_bisect/builds/2933/steps/device_status_check/logs/stdio

Error Log:
==========
Traceback (most recent call last):
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/build/android/buildbot/bb_device_status_check.py", line 258, in blacklisting_recovery
    device.WaitUntilFullyBooted(retries=0)
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 167, in Run
    error_log_func=error_log_func)
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 186, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 158, in _JoinAll
    thread.ReraiseIfException()
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 81, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 160, in <lambda>
    child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/android/device_utils.py", line 515, in WaitUntilFullyBooted
    self.adb.WaitForDevice()
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 723, in WaitForDevice
    self._RunDeviceAdbCmd(['wait-for-device'], timeout, retries)
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 259, in _RunDeviceAdbCmd
    check_error=check_error)
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
    return impl()
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 226, in _RunAdbCmd
    timeout_retry.CurrentTimeoutThreadGroup().GetRemainingTime())
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/utils/cmd_helper.py", line 273, in GetCmdStatusAndOutputWithTimeout
    raise TimeoutError(output.getvalue())
CommandTimeoutError
I   35.081s TimeoutThread-1-for-blacklisting_recovery(0a1409cf)  [host]> /b/build/slave/android_nexus7_perf_bisect/build/src/third_party/android_tools/sdk/platform-tools/adb -s 0a1409cf reboot
E   35.092s blacklisting_recovery(0a1409cf)  Failed to reboot 0a1409cf.
Traceback (most recent call last):
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/build/android/buildbot/bb_device_status_check.py", line 267, in blacklisting_recovery
    device.Reboot(block=False, timeout=5, retries=0)
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 167, in Run
    error_log_func=error_log_func)
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 186, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 158, in _JoinAll
    thread.ReraiseIfException()
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 81, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/utils/timeout_retry.py", line 160, in <lambda>
    child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/android/device_utils.py", line 543, in Reboot
    self.adb.Reboot()
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 774, in Reboot
    self._RunDeviceAdbCmd(cmd, timeout, retries)
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 259, in _RunDeviceAdbCmd
    check_error=check_error)
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
    return impl()
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/build/slave/android_nexus7_perf_bisect/build/src/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 235, in _RunAdbCmd
    args, output, status, device_serial)
AdbCommandFailedError: (device: 0a1409cf) adb reboot: failed with exit status 1 and output:
- error: device offline
Cc: -pras...@chromium.org fmea...@chromium.org
Owner: pras...@chromium.org
Labels: -Infra Infra-Labs
Owner: ----
Status: Available (was: Assigned)
Looks like devices are offline on the following slaves on android_nexus7_perf_bisect builder

build101-b4
build102-b4
build103-b4

Comment 6 by pschm...@google.com, Apr 25 2016

Owner: pschmidt@chromium.org
Status: Fixed (was: Available)
Had to power cycle the n7'S to get them back adb aware.

build101-b4:
List of devices attached
09e0f425	device

build102-b4:
List of devices attached
0a1409cf	device

build103-b4:
List of devices attached
09e0d212	device

Comment 7 by pschm...@google.com, Apr 25 2016

Issue 605918 has been merged into this issue.

Comment 9 by benhenry@google.com, Apr 27 2016

Components: Infra>Labs
Labels: -Infra-Labs
Components: Speed>Bisection

Sign in to add a comment