New issue
Advanced search Search tips

Issue 845564 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: May 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Android
Pri: 2
Type: ----



Sign in to add a comment

Possible issue with device AG86044206I8CGC on chromium.perf/Android One Perf

Project Member Reported by sheriff-...@appspot.gserviceaccount.com, May 22 2018

Issue description

Filed by sheriff-o-matic@appspot.gserviceaccount.com on behalf of sullivan@google.com

Sheriff-o-matic shows that blink_perf.css failing on chromium.perf/Android One Perf

Builders failed on: 
- Android One Perf: 
  https://ci.chromium.org/buildbot/chromium.perf/Android%20One%20Perf

There are two failed builds. Looking at the logs of the first failure, I see a lot of:
https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_One_Perf%2F1564%2F%2B%2Frecipes%2Fsteps%2Fblink_perf.css_on_Android%2F0%2Fstdout
(ERROR) 2018-05-22 10:12:20,470 exceptions.__init__:76  Problem when trying to gather stack trace:
Traceback (most recent call last):
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/core/exceptions.py", line 72, in __init__
    self._is_valid_dump, trace_output = app.GetStackTrace()
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/browser.py", line 237, in GetStackTrace
    return self._browser_backend.GetStackTrace()
  File "/b/swarming/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome/android_browser_backend.py", line 192, in GetStackTrace
    return self.platform_backend.GetStackTrace()
  File "/b/swarming/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/android_platform_backend.py", line 642, in GetStackTrace
    ret = Decorate('UI dump', '\n'.join(self.GetSystemUi().ScreenDump()))
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/app_ui.py", line 212, in ScreenDump
    return self._GetRootUiNode().Dump()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run
    error_log_func=error_log_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 189, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 161, in _JoinAll
    thread.ReraiseIfException()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 84, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda>
    child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/app_ui.py", line 201, in _GetRootUiNode
    check_return=True)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
    return impl()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1112, in RunShellCommand
    output = handle_large_output(cmd, large_output)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1081, in handle_large_output
    return handle_large_command(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1063, in handle_large_command
    return handle_check_return(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1054, in handle_check_return
    return run(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 1050, in run
    return self.adb.Shell(cmd)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 522, in Shell
    output = self._RunDeviceAdbCmd(args, timeout, retries, check_error=False)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 305, in _RunDeviceAdbCmd
    check_error=check_error)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 51, in timeout_retry_wrapper
    return impl()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 282, in _RunAdbCmd
    raise device_errors.DeviceUnreachableError(device_serial)
DeviceUnreachableError: AG86044206I8CGC

Then looking at the second failure:
https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_One_Perf%2F1565%2F%2B%2Frecipes%2Fsteps%2Fblink_perf.css_on_Android%2F0%2Fstdout
ERROR) 2018-05-22 13:37:54,689 device_utils.handle_large_output:1084  Partial output found:
- package:com.android.fmradio
- package:com.google.android.youtube
- package:com.android.providers.telephony
- package:com.google.android.googlequicksearchbox
- package:com.android.providers.calendar
<
s
n
i
p
>
- package:com.google.android.email
- package:com.google.android.music
- package:com.android.printspooler
- package:com.google.android.volta
- package:com.android.dreams.basic
(WARNING) 2018-05-22 13:37:54,689 device_utils.handle_large_output:1085  Attempting to run in large_output mode.
(WARNING) 2018-05-22 13:37:54,689 device_utils.handle_large_output:1086  Use RunShellCommand(..., large_output=True) for shell commands that expect a lot of output.
(INFO) 2018-05-22 13:37:54,689 cmd_helper._ValidateAndLogCommand:160  [host]> /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s AG86044206I8CGC shell '( ( pm list packages )>/data/local/tmp/temp_file-ecfcc4a67910d 2>&1 );echo %$?'
(INFO) 2018-05-22 13:37:54,692 cmd_helper._ValidateAndLogCommand:160  [host]> /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s AG86044206I8CGC shell 'rm -f /data/local/tmp/temp_file-ecfcc4a67910d'
(CRITICAL) 2018-05-22 13:37:54,695 timeout_retry.Run:173  (TimeoutThread-1-for-delete_temporary_file(AG86044206I8CGC)) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['shell', 'rm -f /data/local/tmp/temp_file-ecfcc4a67910d'], check_error=False, retries=2, device_serial=AG86044206I8CGC, timeout=30), attempt 1 of 3: DeviceUnreachableError('AG86044206I8CGC',)
(INFO) 2018-05-22 13:37:54,696 cmd_helper._ValidateAndLogCommand:160  [host]> /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s AG86044206I8CGC shell 'rm -f /data/local/tmp/temp_file-ecfcc4a67910d'
(CRITICAL) 2018-05-22 13:37:54,699 timeout_retry.Run:173  (TimeoutThread-2-for-delete_temporary_file(AG86044206I8CGC)) Exception on _RunAdbCmd(<class 'devil.android.sdk.adb_wrapper.AdbWrapper'>, ['shell', 'rm -f /data/local/tmp/temp_file-ecfcc4a67910d'], check_error=False, retries=2, device_serial=AG86044206I8CGC, timeout=30), attempt 2 of 3: DeviceUnreachableError('AG86044206I8CGC',)
(INFO) 2018-05-22 13:37:54,699 cmd_helper._ValidateAndLogCommand:160  [host]> /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s AG86044206I8CGC shell 'rm -f /data/local/tmp/temp_file-ecfcc4a67910d'
(WARNING) 2018-05-22 13:37:54,702 device_temp_file.delete_temporary_file:66  Failed to delete temporary file /data/local/tmp/temp_file-ecfcc4a67910d: AG86044206I8CGC
(CRITICAL) 2018-05-22 13:37:54,706 timeout_retry.Run:173  (TimeoutThread-1-for-MainThread) Exception on RunShellCommand(AG86044206I8CGC, ['pm', 'list', 'packages'], check_return=True, timeout=30, retries=3), attempt 1 of 4: DeviceUnreachableError('AG86044206I8CGC',)
(INFO) 2018-05-22 13:37:54,706 cmd_helper._ValidateAndLogCommand:160  [host]> /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s AG86044206I8CGC shell '( pm list packages );echo %$?'
(CRITICAL) 2018-05-22 13:37:54,709 timeout_retry.Run:173  (TimeoutThread-2-for-MainThread) Exception on RunShellCommand(AG86044206I8CGC, ['pm', 'list', 'packages'], check_return=True, timeout=30, retries=3), attempt 2 of 4: DeviceUnreachableError('AG86044206I8CGC',)
(INFO) 2018-05-22 13:37:54,709 cmd_helper._ValidateAndLogCommand:160  [host]> /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s AG86044206I8CGC shell '( pm list packages );echo %$?'
(CRITICAL) 2018-05-22 13:37:54,713 timeout_retry.Run:173  (TimeoutThread-3-for-MainThread) Exception on RunShellCommand(AG86044206I8CGC, ['pm', 'list', 'packages'], check_return=True, timeout=30, retries=3), attempt 3 of 4: DeviceUnreachableError('AG86044206I8CGC',)
(INFO) 2018-05-22 13:37:54,713 cmd_helper._ValidateAndLogCommand:160  [host]> /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s AG86044206I8CGC shell '( pm list packages );echo %$?'
(ERROR) 2018-05-22 13:37:54,716 android_browser_finder.FindAllAvailableBrowsers:351  Unable to find browsers on AG86044206I8CGC: AG86044206I8CGC

Can labs check the device?
 
Components: Infra>Labs
Labels: OS-Android

Comment 2 by jo...@google.com, May 22 2018

Owner: jo...@chromium.org
Status: Assigned (was: Available)
Thanks, John!

Also seeing this in speedometer logs: https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_One_Perf%2F1565%2F%2B%2Frecipes%2Fsteps%2Fspeedometer_on_Android%2F0%2Fstdout

DeviceUnreachableError: AG86044206I8CGC

Comment 4 by jo...@chromium.org, May 22 2018

Looks like it had a very unstable USB connection. I swapped out all the old cables on all devices on that host with a newer-spec (more stable) variety. Hopefully this helps things long term.

Comment 5 by jo...@chromium.org, May 24 2018

Status: Fixed (was: Assigned)
Going by a couple of samples, the device health seems more stable (let us know if not)...


Sign in to add a comment