Possible issue with device AG86044206I8CGC on chromium.perf/Android One Perf |
|||
Issue descriptionFiled by sheriff-o-matic@appspot.gserviceaccount.com on behalf of sullivan@google.com Sheriff-o-matic shows that blink_perf.css failing on chromium.perf/Android One Perf Builders failed on: - Android One Perf: https://ci.chromium.org/buildbot/chromium.perf/Android%20One%20Perf There are two failed builds. Looking at the logs of the first failure, I see a lot of: https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_One_Perf%2F1564%2F%2B%2Frecipes%2Fsteps%2Fblink_perf.css_on_Android%2F0%2Fstdout (ERROR) 2018-05-22 10:12:20,470 exceptions.__init__:76 Problem when trying to gather stack trace: Traceback (most recent call last): File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/core/exceptions.py", line 72, in __init__ self._is_valid_dump, trace_output = app.GetStackTrace() File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/browser.py", line 237, in GetStackTrace return self._browser_backend.GetStackTrace() File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function return func(*args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome/android_browser_backend.py", line 192, in GetStackTrace return self.platform_backend.GetStackTrace() File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/android_platform_backend.py", line 642, in GetStackTrace ret = Decorate('UI dump', '\n'.join(self.GetSystemUi().ScreenDump())) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/app_ui.py", line 212, in ScreenDump return self._GetRootUiNode().Dump() File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper retry_if_func=retry_if_func) File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run error_log_func=error_log_func) File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 189, in JoinAll self._JoinAll(watcher, timeout) File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 161, in _JoinAll thread.ReraiseIfException() File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 84, in run self._ret = self._func(*self._args, **self._kwargs) File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda> child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/app_ui.py", line 201, in _GetRootUiNode check_return=True) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper return impl() File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1112, in RunShellCommand output = handle_large_output(cmd, large_output) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1081, in handle_large_output return handle_large_command(cmd) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1063, in handle_large_command return handle_check_return(cmd) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1054, in handle_check_return return run(cmd) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1050, in run return self.adb.Shell(cmd) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 522, in Shell output = self._RunDeviceAdbCmd(args, timeout, retries, check_error=False) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 305, in _RunDeviceAdbCmd check_error=check_error) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper return impl() File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl return f(*args, **kwargs) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 282, in _RunAdbCmd raise device_errors.DeviceUnreachableError(device_serial) DeviceUnreachableError: AG86044206I8CGC Then looking at the second failure: https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_One_Perf%2F1565%2F%2B%2Frecipes%2Fsteps%2Fblink_perf.css_on_Android%2F0%2Fstdout ERROR) 2018-05-22 13:37:54,689 device_utils.handle_large_output:1084 Partial output found: - package:com.android.fmradio - package:com.google.android.youtube - package:com.android.providers.telephony - package:com.google.android.googlequicksearchbox - package:com.android.providers.calendar < s n i p > - package:com.google.android.email - package:com.google.android.music - package:com.android.printspooler - package:com.google.android.volta - package:com.android.dreams.basic (WARNING) 2018-05-22 13:37:54,689 device_utils.handle_large_output:1085 Attempting to run in large_output mode. (WARNING) 2018-05-22 13:37:54,689 device_utils.handle_large_output:1086 Use RunShellCommand(..., large_output=True) for shell commands that expect a lot of output. (INFO) 2018-05-22 13:37:54,689 cmd_helper._ValidateAndLogCommand:160 [host]> /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s AG86044206I8CGC shell '( ( pm list packages )>/data/local/tmp/temp_file-ecfcc4a67910d 2>&1 );echo %$?' (INFO) 2018-05-22 13:37:54,692 cmd_helper._ValidateAndLogCommand:160 [host]> /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s AG86044206I8CGC shell 'rm -f /data/local/tmp/temp_file-ecfcc4a67910d' (CRITICAL) 2018-05-22 13:37:54,695 timeout_retry.Run:173 (TimeoutThread-1-for-delete_temporary_file(AG86044206I8CGC)) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['shell', 'rm -f /data/local/tmp/temp_file-ecfcc4a67910d'], check_error=False, retries=2, device_serial=AG86044206I8CGC, timeout=30), attempt 1 of 3: DeviceUnreachableError('AG86044206I8CGC',) (INFO) 2018-05-22 13:37:54,696 cmd_helper._ValidateAndLogCommand:160 [host]> /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s AG86044206I8CGC shell 'rm -f /data/local/tmp/temp_file-ecfcc4a67910d' (CRITICAL) 2018-05-22 13:37:54,699 timeout_retry.Run:173 (TimeoutThread-2-for-delete_temporary_file(AG86044206I8CGC)) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['shell', 'rm -f /data/local/tmp/temp_file-ecfcc4a67910d'], check_error=False, retries=2, device_serial=AG86044206I8CGC, timeout=30), attempt 2 of 3: DeviceUnreachableError('AG86044206I8CGC',) (INFO) 2018-05-22 13:37:54,699 cmd_helper._ValidateAndLogCommand:160 [host]> /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s AG86044206I8CGC shell 'rm -f /data/local/tmp/temp_file-ecfcc4a67910d' (WARNING) 2018-05-22 13:37:54,702 device_temp_file.delete_temporary_file:66 Failed to delete temporary file /data/local/tmp/temp_file-ecfcc4a67910d: AG86044206I8CGC (CRITICAL) 2018-05-22 13:37:54,706 timeout_retry.Run:173 (TimeoutThread-1-for-MainThread) Exception on RunShellCommand(AG86044206I8CGC, ['pm', 'list', 'packages'], check_return=True, timeout=30, retries=3), attempt 1 of 4: DeviceUnreachableError('AG86044206I8CGC',) (INFO) 2018-05-22 13:37:54,706 cmd_helper._ValidateAndLogCommand:160 [host]> /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s AG86044206I8CGC shell '( pm list packages );echo %$?' (CRITICAL) 2018-05-22 13:37:54,709 timeout_retry.Run:173 (TimeoutThread-2-for-MainThread) Exception on RunShellCommand(AG86044206I8CGC, ['pm', 'list', 'packages'], check_return=True, timeout=30, retries=3), attempt 2 of 4: DeviceUnreachableError('AG86044206I8CGC',) (INFO) 2018-05-22 13:37:54,709 cmd_helper._ValidateAndLogCommand:160 [host]> /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s AG86044206I8CGC shell '( pm list packages );echo %$?' (CRITICAL) 2018-05-22 13:37:54,713 timeout_retry.Run:173 (TimeoutThread-3-for-MainThread) Exception on RunShellCommand(AG86044206I8CGC, ['pm', 'list', 'packages'], check_return=True, timeout=30, retries=3), attempt 3 of 4: DeviceUnreachableError('AG86044206I8CGC',) (INFO) 2018-05-22 13:37:54,713 cmd_helper._ValidateAndLogCommand:160 [host]> /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s AG86044206I8CGC shell '( pm list packages );echo %$?' (ERROR) 2018-05-22 13:37:54,716 android_browser_finder.FindAllAvailableBrowsers:351 Unable to find browsers on AG86044206I8CGC: AG86044206I8CGC Can labs check the device?
,
May 22 2018
,
May 22 2018
Thanks, John! Also seeing this in speedometer logs: https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_One_Perf%2F1565%2F%2B%2Frecipes%2Fsteps%2Fspeedometer_on_Android%2F0%2Fstdout DeviceUnreachableError: AG86044206I8CGC
,
May 22 2018
Looks like it had a very unstable USB connection. I swapped out all the old cables on all devices on that host with a newer-spec (more stable) variety. Hopefully this helps things long term.
,
May 24 2018
Going by a couple of samples, the device health seems more stable (let us know if not)... |
|||
►
Sign in to add a comment |
|||
Comment 1 by sullivan@chromium.org
, May 22 2018Labels: OS-Android