Telemetry attempting to download adb from cloud storage |
||||||||||||
Issue descriptionTelemetry is attempting to download adb from cloud storage on the linux_chromium_rel_ng trybot, which is invalid since this bot doesn't run Android binaries in the first place. See the following failed tryjob: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/214756 and in particular the following three failures: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/214756/steps/context_lost_tests%20on%20NVIDIA%20GPU%20on%20Linux%20%28with%20patch%29%20on%20Linux/logs/stdio https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/214756/steps/hardware_accelerated_feature_tests%20on%20NVIDIA%20GPU%20on%20Linux%20%28with%20patch%29%20on%20Linux/logs/stdio https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/214756/steps/webgl_conformance_tests%20on%20NVIDIA%20GPU%20on%20Linux%20%28with%20patch%29%20on%20Linux/logs/stdio These failed on three different bots in the Swarming pool (build147-m4, build77-m4, and build106-m4). The download attempt is failing with: ---------- Traceback (most recent call last): <module> at /tmp/run1NyNwh/content/test/gpu/run_gpu_test.py:92 retcode = benchmark_runner.main(gpu_project_config.CONFIG) main at /tmp/run1NyNwh/third_party/catapult/telemetry/telemetry/benchmark_runner.py:394 parser = command.CreateParser() CreateParser at /tmp/run1NyNwh/third_party/catapult/telemetry/telemetry/benchmark_runner.py:173 parser = options.CreateParser('%%prog %s %s' % (cls.Name(), cls.usage)) CreateParser at /tmp/run1NyNwh/third_party/catapult/telemetry/telemetry/internal/browser/browser_options.py:104 profiler_choices = profiler_finder.GetAllAvailableProfilers() GetAllAvailableProfilers at /tmp/run1NyNwh/third_party/catapult/telemetry/telemetry/internal/platform/profiler/profiler_finder.py:28 if p.is_supported(browser_type='any')]) is_supported at /tmp/run1NyNwh/third_party/catapult/telemetry/telemetry/internal/platform/profiler/android_traceview_profiler.py:51 return android_browser_finder.CanFindAvailableBrowsers() CanFindAvailableBrowsers at /tmp/run1NyNwh/third_party/catapult/telemetry/telemetry/internal/backends/chrome/android_browser_finder.py:164 return android_device.CanDiscoverDevices() CanDiscoverDevices at /tmp/run1NyNwh/third_party/catapult/telemetry/telemetry/internal/platform/android_device.py:154 if not _HasValidAdb(): _HasValidAdb at /tmp/run1NyNwh/third_party/catapult/telemetry/telemetry/internal/platform/android_device.py:142 adb_path = adb_wrapper.AdbWrapper.GetAdbPath() GetAdbPath at /tmp/run1NyNwh/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py:203 return cls._adb_path.read() read at /tmp/run1NyNwh/third_party/catapult/devil/devil/utils/lazy/weak_constant.py:27 self._val = self._initializer() _FindAdb at /tmp/run1NyNwh/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py:63 return devil_env.config.FetchPath('adb') FetchPath at /tmp/run1NyNwh/third_party/catapult/devil/devil/devil_env.py:131 return self._dm.FetchPath(dependency, GetPlatform(arch, device)) FetchPath at /tmp/run1NyNwh/third_party/catapult/dependency_manager/dependency_manager/manager.py:130 path, _ = self.FetchPathWithVersion(dependency, platform) FetchPathWithVersion at /tmp/run1NyNwh/third_party/catapult/dependency_manager/dependency_manager/manager.py:93 path = dependency_info.GetRemotePath() GetRemotePath at /tmp/run1NyNwh/third_party/catapult/dependency_manager/dependency_manager/dependency_info.py:84 return self._cloud_storage_info.GetRemotePath() GetRemotePath at /tmp/run1NyNwh/third_party/catapult/dependency_manager/dependency_manager/cloud_storage_info.py:80 self._cs_hash) GetIfHashChanged at /tmp/run1NyNwh/third_party/catapult/catapult_base/catapult_base/cloud_storage.py:313 _GetLocked(bucket, cs_path, download_path) _GetLocked at /tmp/run1NyNwh/third_party/catapult/catapult_base/catapult_base/cloud_storage.py:263 _RunCommand(['cp', url, partial_download_path.name]) _RunCommand at /tmp/run1NyNwh/third_party/catapult/catapult_base/catapult_base/cloud_storage.py:166 raise CloudStorageError(stderr) CloudStorageError: Your "Oauth 2.0 User Account" credentials are invalid. For more help, see "gsutil help creds", or re-run the gsutil config command (see "gsutil help config"). Failure: internal_failure. Locals: args : ['/tmp/run1NyNwh/third_party/catapult/third_party/gsutil/gsutil', 'cp', u'gs://chromium-telemetry/binary_dependencies/adb_0c2043552619c8ec8bb5d986ba75703a598611fc', u'/tmp/run1NyNwh/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/tmpA449jd'] disable_cloud_storage_env_val : None gsutil : <subprocess.Popen object at 0x7fadd59ab5d0> gsutil_env : None stderr : 'Your "Oauth 2.0 User Account" credentials are invalid. For more help, see "gsutil help creds", or re-run the gsutil config command (see "gsutil help config").\nFailure: internal_failure.\n' stdout : '' Traceback (most recent call last): File "../../testing/scripts/run_telemetry_benchmark_as_googletest.py", line 69, in main with open(tempfile_name) as f: IOError: [Errno 2] No such file or directory: '/tmp/tmpCyUQZ0/tmpQ8rMkDtelemetry/results.json' Running ['/usr/bin/python', '../../content/test/gpu/run_gpu_test.py', 'context_lost', '--show-stdout', '--browser=release', '-v', '--extra-browser-args=--enable-logging=stderr --js-flags=--expose-gc', '--output-dir', '/tmp/tmpCyUQZ0/tmpQ8rMkDtelemetry', '--output-format=json'] Command ['/usr/bin/python', '../../content/test/gpu/run_gpu_test.py', 'context_lost', '--show-stdout', '--browser=release', '-v', '--extra-browser-args=--enable-logging=stderr --js-flags=--expose-gc', '--output-dir', '/tmp/tmpCyUQZ0/tmpQ8rMkDtelemetry', '--output-format=json'] returned exit code 1 ---------- The issue here isn't that the machine doesn't have the right credentials; it's that Telemetry should not be trying to download adb in the first place. (The command line argument "--browser=release" should completely short-circuit any detection of connected Android devices in Telemetry.) Marking P1 as this has been seen on the CQ. Ned, could you please find someone to fix this ASAP?
,
Apr 19 2016
Even if telemetry doesn't download adb, it would still download other binaries specified in https://code.google.com/p/chromium/codesearch#chromium/src/third_party/catapult/telemetry/telemetry/internal/binary_dependencies.json Fixing telemetry so that we have an extra step that fetch all the dependent binaries before running swarming test is not a simple task, and shouldn't block this P1 bug. "chromium-telemetry" is a public bucket, so I am not sure why it raises the credential error here.
,
Apr 19 2016
Reassign to kbr@ to triage this bug. For a quick fix, I would say fixing the credential error when telemetry try to download from the public cloud storage.
,
Apr 20 2016
I'm sorry, I do not have the time to triage why these credentials do not exist on these bots. The fact remains that when --browser=release is specified on Linux, Telemetry should not be attempting to download adb. Please fix this. The remaining binary dependencies should be bundled into the isolates in the Chromium build.
,
Apr 20 2016
At this point this doesn't seem to be affecting the commit queue: https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng?numbuilds=200 There are no failing builds due to GPU tests. Downgrading to P2 as a consequence. We should figure out what the best steps are to conclusively solve the problem of downloads from cloud storage on the bots. Hacks to specifically avoid running adb when using a desktop Linux browser are probably not needed.
,
Apr 20 2016
Kari: can you drive the effort of fetching all the binaries before zipping & send all the files to swarming?
,
Apr 20 2016
There is more than one issue here. 1) Why are the bots failing to fetch from the public bucket? This seems like a bug that should be investigated in and of itself, and is likely a credentials or cloud storage issue. 2) We should pre-fetch dependencies before running the tests on bots. Ned looked into this a while back, and we realized that it's hard. This should be done as part of the swarming work, which I believe is slated for next quarter. 3) I agree that we shouldn't need to download adb if we're running on a Linux box, but we currently use it to determine whether we are able to run on android. (--browser=reference doesn't refer specifically to the desktop reference browser. We add the --device flag to specify an android device to run on instead.) The browser_finder should be refactored so that we are only looking for browsers on the platform specified, or the local platform if no devices are specified. This bug appears to be specific to #1. If it happens again, we should ask infra to look into the credentials/network access on the bots in question. I'd be happy to help with 2 or 3. I might get to #3 but won't get to #2 this quarter unless it becomes a higher priority. They should each be tracked in separate bugs since the scope of them is larger than this specific issue.
,
Apr 20 2016
From comment #7: #2 is covered in https://crbug.com/591173 which is assigned to me. #3 we should hijack https://crbug.com/435723 for this work. It is currently unassigned. Can someone on Infra comment on if there is anything else we can do for #1 here?
,
Apr 20 2016
,
Apr 26 2016
,
Apr 27 2016
,
May 19 2016
Why is this bug in the trooper queue? I don't see what a trooper can do here. Removing the label.
,
Sep 6 2016
|
||||||||||||
►
Sign in to add a comment |
||||||||||||
Comment 1 by nedngu...@google.com
, Apr 19 2016