Filed by sheriff-o-matic@appspot.gserviceaccount.com on behalf of johnchen@chromium.org

performance_webview_test_suite failing on chromium.perf/Android Nexus5X WebView Perf

Builders failed on: 
- Android Nexus5X WebView Perf: 
  https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/Android%20Nexus5X%20WebView%20Perf

Occasionally, a shard or two in performance_webview_test_suite would fail while acquiring device. For example:
* https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/Android%20Nexus5X%20WebView%20Perf/4320 shard 13
* https://ci.chromium.org/p/chrome/builders/luci.chrome.ci/Android%20Nexus5X%20WebView%20Perf/4303 shards 5 and 6

Here is a typical output from a failed run:
I    0.035s _FindAdb  Downloading gs://chromium-telemetry/binary_dependencies/adb_8bd43e3930f6eec643d5dc64cab9e5bb4ddf4909 to /b/swarming/w/ir/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb
I    1.034s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (1.0s)
I    2.035s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (2.0s)
I    3.035s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (3.0s)
I    4.035s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (4.0s)
I    5.036s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (5.0s)
I    6.036s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (6.0s)
I    7.036s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (7.0s)
I    8.037s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (8.0s)
I    9.037s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (9.0s)
I   10.037s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (10.0s)
I   11.038s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (11.0s)
I   12.038s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (12.0s)
I   13.039s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (13.0s)
I   14.039s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (14.0s)
I   15.039s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (15.0s)
I   16.040s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (16.0s)
I   17.040s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (17.0s)
I   18.041s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (18.0s)
I   19.041s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (19.0s)
I   20.041s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (20.0s)
I   21.041s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (21.0s)
I   22.042s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (22.0s)
I   23.042s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (23.0s)
I   24.042s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (24.0s)
I   25.043s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (25.0s)
I   26.043s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (26.0s)
I   27.043s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (27.0s)
I   28.044s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (28.0s)
I   29.044s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (29.0s)
I   30.045s TimeoutThread-1-for-MainThread  condition '<lambda>' not met (30.0s)
C   30.081s Main  Timed out. Dumping threads.
I   31.083s TimeoutThread-2-for-MainThread  condition '<lambda>' not met (1.0s)
I   32.083s TimeoutThread-2-for-MainThread  condition '<lambda>' not met (2.0s)
I   33.084s TimeoutThread-2-for-MainThread  condition '<lambda>' not met (3.0s)
I   34.084s TimeoutThread-2-for-MainThread  condition '<lambda>' not met (4.0s)
I   35.084s TimeoutThread-2-for-MainThread  condition '<lambda>' not met (5.0s)
I   36.084s TimeoutThread-2-for-MainThread  condition '<lambda>' not met (6.0s)
I   37.085s TimeoutThread-2-for-MainThread  condition '<lambda>' not met (7.0s)
I   38.085s TimeoutThread-2-for-MainThread  condition '<lambda>' not met (8.0s)
I   39.086s TimeoutThread-2-for-MainThread  condition '<lambda>' not met (9.0s)
I   40.086s TimeoutThread-2-for-MainThread  condition '<lambda>' not met (10.0s)
I   41.086s TimeoutThread-2-for-MainThread  condition '<lambda>' not met (11.0s)
I   42.087s TimeoutThread-2-for-MainThread  condition '<lambda>' not met (12.0s)
I   42.102s TimeoutThread-2-for-MainThread  condition '<lambda>' met (12.0s)
Traceback (most recent call last):
  File "../../third_party/catapult/devil/devil/android/tools/system_app.py", line 239, in <module>
    sys.exit(main(sys.argv[1:]))
  File "../../third_party/catapult/devil/devil/android/tools/system_app.py", line 229, in main
    devices = script_common.GetDevices(args.devices, args.blacklist_file)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/tools/script_common.py", line 45, in GetDevices
    devices = device_utils.DeviceUtils.HealthyDevices(blacklist_file)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 2937, in HealthyDevices
    return _get_devices()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/device_utils.py", line 2910, in _get_devices
    for adb in adb_wrapper.AdbWrapper.Devices():
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 398, in Devices
    retries=retries)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 418, in _RawDevices
    output = cls._RunAdbCmd(cmd, timeout=timeout, retries=retries)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 57, in timeout_retry_wrapper
    retry_if_func=retry_if_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 158, in Run
    error_log_func=error_log_func)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 198, in JoinAll
    self._JoinAll(watcher, timeout)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 170, in _JoinAll
    thread.ReraiseIfException()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/timeout_retry.py", line 151, in <lambda>
    child_thread = reraiser_thread.ReraiserThread(lambda: func(*args, **kwargs),
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/decorators.py", line 47, in impl
    return f(*args, **kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 276, in _RunAdbCmd
    cls._BuildAdbCmd(args, device_serial, cpu_affinity=cpu_affinity),
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 255, in _BuildAdbCmd
    cmd.append(cls.GetAdbPath())
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 243, in GetAdbPath
    return cls._adb_path.read()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/lazy/weak_constant.py", line 39, in read
    self._val = initializer_thread.GetReturnValue()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 86, in GetReturnValue
    self.ReraiseIfException()
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/utils/reraiser_thread.py", line 93, in run
    self._ret = self._func(*self._args, **self._kwargs)
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py", line 84, in _FindAdb
    return devil_env.config.FetchPath('adb')
  File "/b/swarming/w/ir/third_party/catapult/devil/devil/devil_env.py", line 175, in FetchPath
    return self._dm.FetchPath(dependency, GetPlatform(arch, device))
  File "/b/swarming/w/ir/third_party/catapult/dependency_manager/dependency_manager/manager.py", line 130, in FetchPath
    path, _ = self.FetchPathWithVersion(dependency, platform)
  File "/b/swarming/w/ir/third_party/catapult/dependency_manager/dependency_manager/manager.py", line 93, in FetchPathWithVersion
    path = dependency_info.GetRemotePath()
  File "/b/swarming/w/ir/third_party/catapult/dependency_manager/dependency_manager/dependency_info.py", line 84, in GetRemotePath
    return self._cloud_storage_info.GetRemotePath()
  File "/b/swarming/w/ir/third_party/catapult/dependency_manager/dependency_manager/cloud_storage_info.py", line 80, in GetRemotePath
    self._cs_hash)
  File "/b/swarming/w/ir/third_party/catapult/common/py_utils/py_utils/cloud_storage.py", line 400, in GetIfHashChanged
    with _FileLock(download_path):
  File "/b/swarming/w/ir/.swarming_module/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/b/swarming/w/ir/third_party/catapult/common/py_utils/py_utils/cloud_storage.py", line 272, in _FileLock
    LOCK_ACQUISITION_TIMEOUT)
  File "/b/swarming/w/ir/third_party/catapult/common/py_utils/py_utils/__init__.py", line 150, in WaitFor
    (timeout, GetConditionString()))
py_utils.TimeoutException: Timed out while waiting 10s for py_utils.WaitFor(lambda: _AttemptPseudoLockAcquisition(pseudo_lock_path,
                                                         pseudo_lock_fd_return),
                   LOCK_ACQUISITION_TIMEOUT).