New issue
Advanced search Search tips

Issue 839378 link

Starred by 2 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: ----



Sign in to add a comment

file lock timeout in blink_perf.bindings on chromium.perf/Android Nexus5X WebView Perf

Project Member Reported by sheriff-...@appspot.gserviceaccount.com, May 3 2018

Issue description

Filed by sheriff-o-matic@appspot.gserviceaccount.com on behalf of sullivan@google.com

Builders failed on: 
- Android Nexus5X WebView Perf: 
  https://ci.chromium.org/buildbot/chromium.perf/Android%20Nexus5X%20WebView%20Perf

Only failing on the one test, but looks like a telemetry issue. Assigning to Ned for triage.

Seeing this in the logs for both builds #1808 and 1809:
https://logs.chromium.org/v/?s=chrome%2Fbb%2Fchromium.perf%2FAndroid_Nexus5X_WebView_Perf%2F1809%2F%2B%2Frecipes%2Fsteps%2Fblink_perf.bindings_on_Android%2F0%2Fstdout#

Traceback (most recent call last):
  File "../../third_party/catapult/devil/devil/android/tools/system_app.py", line 218, in <module>
    sys.exit(main(sys.argv[1:]))
  File "../../third_party/catapult/devil/devil/android/tools/system_app.py", line 208, in main
    devices = script_common.GetDevices(args.devices, args.blacklist_file)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/tools/script_common.py", line 45, in GetDevices
    devices = device_utils.DeviceUtils.HealthyDevices(blacklist_file)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 2815, in HealthyDevices
    return _get_devices()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 2804, in _get_devices
    for adb in adb_wrapper.AdbWrapper.Devices():
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 385, in Devices
    retries=retries)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 405, in _RawDevices
    output = cls._RunAdbCmd(cmd, timeout=timeout, retries=retries)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run
    error_log_func=error_log_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 189, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 161, in _JoinAll
    thread.ReraiseIfException()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 84, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda>
    child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 262, in _RunAdbCmd
    cls._BuildAdbCmd(args, device_serial, cpu_affinity=cpu_affinity),
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 248, in _BuildAdbCmd
    cmd.append(cls.GetAdbPath())
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 236, in GetAdbPath
    return cls._adb_path.read()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/lazy/weak_constant.py", line 39, in read
    self._val = initializer_thread.GetReturnValue()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 77, in GetReturnValue
    self.ReraiseIfException()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 84, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 81, in _FindAdb
    return devil_env.config.FetchPath('adb')
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/devil_env.py", line 175, in FetchPath
    return self._dm.FetchPath(dependency, GetPlatform(arch, device))
  File "/b/swarming/w/ir/third_party/catapult/dependency_manager/dependency_manager/manager.py", line 130, in FetchPath
    path, _ = self.FetchPathWithVersion(dependency, platform)
  File "/b/swarming/w/ir/third_party/catapult/dependency_manager/dependency_manager/manager.py", line 93, in FetchPathWithVersion
    path = dependency_info.GetRemotePath()
  File "/b/swarming/w/ir/third_party/catapult/dependency_manager/dependency_manager/dependency_info.py", line 84, in GetRemotePath
    return self._cloud_storage_info.GetRemotePath()
  File "/b/swarming/w/ir/third_party/catapult/dependency_manager/dependency_manager/cloud_storage_info.py", line 80, in GetRemotePath
    self._cs_hash)
  File "/b/swarming/w/ir/third_party/catapult/common/py_utils/py_utils/cloud_storage.py", line 395, in GetIfHashChanged
    with _FileLock(download_path):
  File "/b/swarming/w/ir/.swarming_module/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/b/swarming/w/ir/third_party/catapult/common/py_utils/py_utils/cloud_storage.py", line 267, in _FileLock
    LOCK_ACQUISITION_TIMEOUT)
  File "/b/swarming/w/ir/third_party/catapult/common/py_utils/py_utils/__init__.py", line 148, in WaitFor
    (timeout, GetConditionString()))
py_utils.TimeoutException: Timed out while waiting 10s for py_utils.WaitFor(lambda: _AttemptPseudoLockAcquisition(pseudo_lock_path,
                                                         pseudo_lock_fd_return),
                   LOCK_ACQUISITION_TIMEOUT).
 
Components: Speed>Telemetry
Owner: nednguyen@chromium.org
Status: Assigned (was: Available)
Cc: perezju@chromium.org
Owner: jbudorick@chromium.org
John: can you help triaging this? This is the problem with devil trying to fetch adb 

Command: /b/swarming/w/ir/.swarming_module_cache/vpython/09eff0/bin/python ../../third_party/catapult/devil/devil/android/tools/system_app.py remove --package com.android.webview com.google.android.webview -v -- ../../testing/scripts/run_telemetry_benchmark_as_googletest.py ../../tools/perf/run_benchmark blink_perf.bindings -v --upload-results --browser=android-webview --output-format=histograms --webview-embedder-apk=../../out/Release/apks/SystemWebViewShell.apk --isolated-script-test-output=/b/swarming/w/io0cX0mX/output.json --isolated-script-test-perf-output=/b/swarming/w/io0cX0mX/perftest-output.json

Comment 4 by benhenry@google.com, Jan 16 (6 days ago)

Components: Test>Telemetry

Comment 5 by benhenry@google.com, Jan 16 (6 days ago)

Components: -Speed>Telemetry

Sign in to add a comment