blink_perf.shadow_dom, blink_perf.shadow_dom, power.typical_10_mobile, smoothness.gpu_rasterization.polymer, on chromium.perf/Android Nexus6 WebView Perf |
|||
Issue descriptionFiled by sheriff-o-matic@appspot.gserviceaccount.com on behalf of eyaich@google.com blink_perf.shadow_dom failing on chromium.perf/Android Nexus6 WebView Perf Builders failed on: - Android Nexus6 WebView Perf: https://build.chromium.org/p/chromium.perf/builders/Android%20Nexus6%20WebView%20Perf
,
Feb 26 2018
From:
File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 2693, in HealthyDevices
return _get_devices()
Looks like it's timing out trying to list the healthy devices. John, could you triage?
,
Feb 26 2018
Interesting -- it looks like the first try is timing out on downloading adb from gs, then subsequent retries fail because the first is still holding the lock.
,
Mar 5 2018
,
Mar 6 2018
The following revision refers to this bug: https://chromium.googlesource.com/catapult/+/69ef1a9b0080d650a7e2b70f344ba8ff07d125b3 commit 69ef1a9b0080d650a7e2b70f344ba8ff07d125b3 Author: John Budorick <jbudorick@chromium.org> Date: Tue Mar 06 18:23:53 2018 [devil] Initialize WeakConstant values on their own thread. Bug: chromium:816469 Change-Id: Ib0e8cfce19caba804af2f4861788e0364f9264d1 Reviewed-on: https://chromium-review.googlesource.com/949304 Commit-Queue: John Budorick <jbudorick@chromium.org> Reviewed-by: Juan Antonio Navarro Pérez <perezju@chromium.org> [add] https://crrev.com/69ef1a9b0080d650a7e2b70f344ba8ff07d125b3/devil/devil/utils/lazy/weak_constant_test.py [modify] https://crrev.com/69ef1a9b0080d650a7e2b70f344ba8ff07d125b3/devil/devil/utils/timeout_retry.py [modify] https://crrev.com/69ef1a9b0080d650a7e2b70f344ba8ff07d125b3/devil/devil/utils/lazy/weak_constant.py [modify] https://crrev.com/69ef1a9b0080d650a7e2b70f344ba8ff07d125b3/devil/devil/utils/reraiser_thread.py |
|||
►
Sign in to add a comment |
|||
Comment 1 by eyaich@chromium.org
, Feb 26 2018Summary: blink_perf.shadow_dom, blink_perf.shadow_dom, power.typical_10_mobile, smoothness.gpu_rasterization.polymer, on chromium.perf/Android Nexus6 WebView Perf (was: blink_perf.shadow_dom failing on chromium.perf/Android Nexus6 WebView Perf)
All of these tests are failing intermittently with this failure. Not sure if there is anything we can do about this flaky device failure: Additional test environment: CHROME_DEVEL_SANDBOX=/opt/chromium/chrome_sandbox CHROME_HEADLESS=1 LANG=en_US.UTF-8 Command: /usr/bin/python ../../third_party/catapult/devil/devil/android/tools/system_app.py remove --package com.android.webview com.google.android.webview -v -- ../../testing/scripts/run_telemetry_benchmark_as_googletest.py ../../tools/perf/run_benchmark power.typical_10_mobile -v --upload-results --browser=android-webview --output-format=chartjson --webview-embedder-apk=../../out/Release/apks/SystemWebViewShell.apk --isolated-script-test-output=/b/swarming/w/iohQDD5Z/output.json --isolated-script-test-perf-output=/b/swarming/w/iohQDD5Z/perftest-output.json I 0.004s TimeoutThread-1-for-MainThread Downloading gs://chromium-telemetry/binary_dependencies/adb_8bd43e3930f6eec643d5dc64cab9e5bb4ddf4909 to /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb C 30.036s Main Timed out. Dumping threads. C 30.036s Main ******************************************************************************** C 30.037s Main Stack dump for thread 'TimeoutThread-1-for-MainThread' C 30.037s Main ******************************************************************************** C 30.699s Main File: "/usr/lib/python2.7/threading.py", line 774, in __bootstrap C 30.699s Main self.__bootstrap_inner() C 30.699s Main File: "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner C 30.700s Main self.run() C 30.700s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 81, in run C 30.700s Main self._ret = self._func(*self._args, **self._kwargs) C 30.700s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 150, in <lambda> C 30.700s Main child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), C 30.700s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl C 30.700s Main return f(*args, **kwargs) C 30.700s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 260, in _RunAdbCmd C 30.700s Main cls._BuildAdbCmd(args, device_serial, cpu_affinity=cpu_affinity), C 30.700s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 246, in _BuildAdbCmd C 30.700s Main cmd.append(cls.GetAdbPath()) C 30.700s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 234, in GetAdbPath C 30.700s Main return cls._adb_path.read() C 30.700s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/lazy/weak_constant.py", line 27, in read C 30.700s Main self._val = self._initializer() C 30.700s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 79, in _FindAdb C 30.700s Main return devil_env.config.FetchPath('adb') C 30.700s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/devil_env.py", line 175, in FetchPath C 30.700s Main return self._dm.FetchPath(dependency, GetPlatform(arch, device)) C 30.700s Main File: "/b/swarming/w/ir/third_party/catapult/dependency_manager/dependency_manager/manager.py", line 130, in FetchPath C 30.700s Main path, _ = self.FetchPathWithVersion(dependency, platform) C 30.700s Main File: "/b/swarming/w/ir/third_party/catapult/dependency_manager/dependency_manager/manager.py", line 93, in FetchPathWithVersion C 30.700s Main path = dependency_info.GetRemotePath() C 30.700s Main File: "/b/swarming/w/ir/third_party/catapult/dependency_manager/dependency_manager/dependency_info.py", line 84, in GetRemotePath C 30.700s Main return self._cloud_storage_info.GetRemotePath() C 30.700s Main File: "/b/swarming/w/ir/third_party/catapult/dependency_manager/dependency_manager/cloud_storage_info.py", line 80, in GetRemotePath C 30.700s Main self._cs_hash) C 30.700s Main File: "/b/swarming/w/ir/third_party/catapult/common/py_utils/py_utils/cloud_storage.py", line 399, in GetIfHashChanged C 30.700s Main _GetLocked(bucket, cs_path, download_path) C 30.701s Main File: "/b/swarming/w/ir/third_party/catapult/common/py_utils/py_utils/cloud_storage.py", line 349, in _GetLocked C 30.701s Main _RunCommand(['cp', url, partial_download_path.name]) C 30.701s Main File: "/b/swarming/w/ir/third_party/catapult/common/py_utils/py_utils/cloud_storage.py", line 151, in _RunCommand C 30.701s Main stdout, stderr = gsutil.communicate() C 30.701s Main File: "/usr/lib/python2.7/subprocess.py", line 800, in communicate C 30.701s Main return self._communicate(input) C 30.701s Main File: "/usr/lib/python2.7/subprocess.py", line 1417, in _communicate C 30.701s Main stdout, stderr = self._communicate_with_poll(input) C 30.701s Main File: "/usr/lib/python2.7/subprocess.py", line 1471, in _communicate_with_poll C 30.701s Main ready = poller.poll() C 30.701s Main ******************************************************************************** C 60.737s Main Timed out. Dumping threads. C 60.737s Main ******************************************************************************** C 60.737s Main Stack dump for thread 'TimeoutThread-2-for-MainThread' C 60.737s Main ******************************************************************************** C 60.737s Main File: "/usr/lib/python2.7/threading.py", line 774, in __bootstrap C 60.737s Main self.__bootstrap_inner() C 60.737s Main File: "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner C 60.737s Main self.run() C 60.737s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 81, in run C 60.737s Main self._ret = self._func(*self._args, **self._kwargs) C 60.737s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 150, in <lambda> C 60.737s Main child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), C 60.737s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl C 60.737s Main return f(*args, **kwargs) C 60.737s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 260, in _RunAdbCmd C 60.737s Main cls._BuildAdbCmd(args, device_serial, cpu_affinity=cpu_affinity), C 60.737s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 246, in _BuildAdbCmd C 60.737s Main cmd.append(cls.GetAdbPath()) C 60.737s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 234, in GetAdbPath C 60.737s Main return cls._adb_path.read() C 60.738s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/lazy/weak_constant.py", line 25, in read C 60.738s Main with self._lock: C 60.738s Main ******************************************************************************** C 90.773s Main Timed out. Dumping threads. C 90.773s Main ******************************************************************************** C 90.773s Main Stack dump for thread 'TimeoutThread-3-for-MainThread' C 90.773s Main ******************************************************************************** C 90.773s Main File: "/usr/lib/python2.7/threading.py", line 774, in __bootstrap C 90.774s Main self.__bootstrap_inner() C 90.774s Main File: "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner C 90.774s Main self.run() C 90.774s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 81, in run C 90.774s Main self._ret = self._func(*self._args, **self._kwargs) C 90.774s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 150, in <lambda> C 90.774s Main child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs), C 90.774s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl C 90.774s Main return f(*args, **kwargs) C 90.774s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 260, in _RunAdbCmd C 90.774s Main cls._BuildAdbCmd(args, device_serial, cpu_affinity=cpu_affinity), C 90.774s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 246, in _BuildAdbCmd C 90.774s Main cmd.append(cls.GetAdbPath()) C 90.774s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 234, in GetAdbPath C 90.774s Main return cls._adb_path.read() C 90.774s Main File: "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/lazy/weak_constant.py", line 25, in read C 90.774s Main with self._lock: C 90.774s Main ******************************************************************************** Traceback (most recent call last): File "../../third_party/catapult/devil/devil/android/tools/system_app.py", line 218, in <module> sys.exit(main(sys.argv[1:])) File "../../third_party/catapult/devil/devil/android/tools/system_app.py", line 208, in main devices = script_common.GetDevices(args.devices, args.blacklist_file) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/tools/script_common.py", line 45, in GetDevices devices = device_utils.DeviceUtils.HealthyDevices(blacklist_file) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 2693, in HealthyDevices return _get_devices() File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 2682, in _get_devices for adb in adb_wrapper.AdbWrapper.Devices(): File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 383, in Devices retries=retries) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 403, in _RawDevices output = cls._RunAdbCmd(cmd, timeout=timeout, retries=retries) File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper retry_if_func=retry_if_func) File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 157, in Run error_log_func=error_log_func) File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 186, in JoinAll self._JoinAll(watcher, timeout) File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 151, in _JoinAll (len(alive_threads), len(self._threads))) devil.android.device_errors.CommandTimeoutError: Timed out waiting for 1 of 1 threads.