New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 604904 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner:
Last visit > 30 days ago
Closed: Sep 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 2
Type: Bug



Sign in to add a comment

Telemetry attempting to download adb from cloud storage

Project Member Reported by kbr@chromium.org, Apr 19 2016

Issue description

Telemetry is attempting to download adb from cloud storage on the linux_chromium_rel_ng trybot, which is invalid since this bot doesn't run Android binaries in the first place.

See the following failed tryjob:
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/214756

and in particular the following three failures:

https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/214756/steps/context_lost_tests%20on%20NVIDIA%20GPU%20on%20Linux%20%28with%20patch%29%20on%20Linux/logs/stdio
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/214756/steps/hardware_accelerated_feature_tests%20on%20NVIDIA%20GPU%20on%20Linux%20%28with%20patch%29%20on%20Linux/logs/stdio
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/214756/steps/webgl_conformance_tests%20on%20NVIDIA%20GPU%20on%20Linux%20%28with%20patch%29%20on%20Linux/logs/stdio

These failed on three different bots in the Swarming pool (build147-m4, build77-m4, and build106-m4).

The download attempt is failing with:

----------
Traceback (most recent call last):
  <module> at /tmp/run1NyNwh/content/test/gpu/run_gpu_test.py:92
    retcode = benchmark_runner.main(gpu_project_config.CONFIG)
  main at /tmp/run1NyNwh/third_party/catapult/telemetry/telemetry/benchmark_runner.py:394
    parser = command.CreateParser()
  CreateParser at /tmp/run1NyNwh/third_party/catapult/telemetry/telemetry/benchmark_runner.py:173
    parser = options.CreateParser('%%prog %s %s' % (cls.Name(), cls.usage))
  CreateParser at /tmp/run1NyNwh/third_party/catapult/telemetry/telemetry/internal/browser/browser_options.py:104
    profiler_choices = profiler_finder.GetAllAvailableProfilers()
  GetAllAvailableProfilers at /tmp/run1NyNwh/third_party/catapult/telemetry/telemetry/internal/platform/profiler/profiler_finder.py:28
    if p.is_supported(browser_type='any')])
  is_supported at /tmp/run1NyNwh/third_party/catapult/telemetry/telemetry/internal/platform/profiler/android_traceview_profiler.py:51
    return android_browser_finder.CanFindAvailableBrowsers()
  CanFindAvailableBrowsers at /tmp/run1NyNwh/third_party/catapult/telemetry/telemetry/internal/backends/chrome/android_browser_finder.py:164
    return android_device.CanDiscoverDevices()
  CanDiscoverDevices at /tmp/run1NyNwh/third_party/catapult/telemetry/telemetry/internal/platform/android_device.py:154
    if not _HasValidAdb():
  _HasValidAdb at /tmp/run1NyNwh/third_party/catapult/telemetry/telemetry/internal/platform/android_device.py:142
    adb_path = adb_wrapper.AdbWrapper.GetAdbPath()
  GetAdbPath at /tmp/run1NyNwh/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py:203
    return cls._adb_path.read()
  read at /tmp/run1NyNwh/third_party/catapult/devil/devil/utils/lazy/weak_constant.py:27
    self._val = self._initializer()
  _FindAdb at /tmp/run1NyNwh/third_party/catapult/devil/devil/android/sdk/adb_wrapper.py:63
    return devil_env.config.FetchPath('adb')
  FetchPath at /tmp/run1NyNwh/third_party/catapult/devil/devil/devil_env.py:131
    return self._dm.FetchPath(dependency, GetPlatform(arch, device))
  FetchPath at /tmp/run1NyNwh/third_party/catapult/dependency_manager/dependency_manager/manager.py:130
    path, _ = self.FetchPathWithVersion(dependency, platform)
  FetchPathWithVersion at /tmp/run1NyNwh/third_party/catapult/dependency_manager/dependency_manager/manager.py:93
    path = dependency_info.GetRemotePath()
  GetRemotePath at /tmp/run1NyNwh/third_party/catapult/dependency_manager/dependency_manager/dependency_info.py:84
    return self._cloud_storage_info.GetRemotePath()
  GetRemotePath at /tmp/run1NyNwh/third_party/catapult/dependency_manager/dependency_manager/cloud_storage_info.py:80
    self._cs_hash)
  GetIfHashChanged at /tmp/run1NyNwh/third_party/catapult/catapult_base/catapult_base/cloud_storage.py:313
    _GetLocked(bucket, cs_path, download_path)
  _GetLocked at /tmp/run1NyNwh/third_party/catapult/catapult_base/catapult_base/cloud_storage.py:263
    _RunCommand(['cp', url, partial_download_path.name])
  _RunCommand at /tmp/run1NyNwh/third_party/catapult/catapult_base/catapult_base/cloud_storage.py:166
    raise CloudStorageError(stderr)
CloudStorageError: Your "Oauth 2.0 User Account" credentials are invalid. For more help, see "gsutil help creds", or re-run the gsutil config command (see "gsutil help config").
Failure: internal_failure.

Locals:
  args                          : ['/tmp/run1NyNwh/third_party/catapult/third_party/gsutil/gsutil', 'cp', u'gs://chromium-telemetry/binary_dependencies/adb_0c2043552619c8ec8bb5d986ba75703a598611fc', u'/tmp/run1NyNwh/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/tmpA449jd']
  disable_cloud_storage_env_val : None
  gsutil                        : <subprocess.Popen object at 0x7fadd59ab5d0>
  gsutil_env                    : None
  stderr                        : 'Your "Oauth 2.0 User Account" credentials are invalid. For more help, see "gsutil help creds", or re-run the gsutil config command (see "gsutil help config").\nFailure: internal_failure.\n'
  stdout                        : ''

Traceback (most recent call last):
  File "../../testing/scripts/run_telemetry_benchmark_as_googletest.py", line 69, in main
    with open(tempfile_name) as f:
IOError: [Errno 2] No such file or directory: '/tmp/tmpCyUQZ0/tmpQ8rMkDtelemetry/results.json'
Running ['/usr/bin/python', '../../content/test/gpu/run_gpu_test.py', 'context_lost', '--show-stdout', '--browser=release', '-v', '--extra-browser-args=--enable-logging=stderr --js-flags=--expose-gc', '--output-dir', '/tmp/tmpCyUQZ0/tmpQ8rMkDtelemetry', '--output-format=json']
Command ['/usr/bin/python', '../../content/test/gpu/run_gpu_test.py', 'context_lost', '--show-stdout', '--browser=release', '-v', '--extra-browser-args=--enable-logging=stderr --js-flags=--expose-gc', '--output-dir', '/tmp/tmpCyUQZ0/tmpQ8rMkDtelemetry', '--output-format=json'] returned exit code 1
----------

The issue here isn't that the machine doesn't have the right credentials; it's that Telemetry should not be trying to download adb in the first place. (The command line argument "--browser=release" should completely short-circuit any detection of connected Android devices in Telemetry.)

Marking P1 as this has been seen on the CQ.

Ned, could you please find someone to fix this ASAP?

 
Cc: eakuefner@chromium.org
Even if telemetry doesn't download adb, it would still download other binaries specified in https://code.google.com/p/chromium/codesearch#chromium/src/third_party/catapult/telemetry/telemetry/internal/binary_dependencies.json

Fixing telemetry so that we have an extra step that fetch all the dependent binaries before running swarming test is not a simple task, and shouldn't block this P1 bug.


"chromium-telemetry" is a public bucket, so I am not sure why it raises the credential error here.
Owner: kbr@chromium.org
Status: Untriaged (was: Assigned)
Reassign to kbr@ to triage this bug. For a quick fix, I would say fixing the credential error when telemetry try to download from the public cloud storage.

Comment 4 by kbr@chromium.org, Apr 20 2016

Owner: nedngu...@google.com
Status: Assigned (was: Untriaged)
I'm sorry, I do not have the time to triage why these credentials do not exist on these bots.

The fact remains that when --browser=release is specified on Linux, Telemetry should not be attempting to download adb. Please fix this.

The remaining binary dependencies should be bundled into the isolates in the Chromium build.

Comment 5 by kbr@chromium.org, Apr 20 2016

Labels: -Pri-1 Pri-2
At this point this doesn't seem to be affecting the commit queue:
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng?numbuilds=200

There are no failing builds due to GPU tests.

Downgrading to P2 as a consequence. We should figure out what the best steps are to conclusively solve the problem of downloads from cloud storage on the bots. Hacks to specifically avoid running adb when using a desktop Linux browser are probably not needed.

Cc: -aiolos@chromium.org nedngu...@google.com
Owner: aiolos@chromium.org
Kari: can you drive the effort of fetching all the binaries before zipping & send all the files to swarming?

Comment 7 by aiolos@chromium.org, Apr 20 2016

Cc: sullivan@chromium.org
There is more than one issue here.

1) Why are the bots failing to fetch from the public bucket? This seems like a bug that should be investigated in and of itself, and is likely a credentials or cloud storage issue.

2) We should pre-fetch dependencies before running the tests on bots. Ned looked into this a while back, and we realized that it's hard. This should be done as part of the swarming work, which I believe is slated for next quarter.

3) I agree that we shouldn't need to download adb if we're running on a Linux box, but we currently use it to determine whether we are able to run on android. (--browser=reference doesn't refer specifically to the desktop reference browser. We add the --device flag to specify an android device to run on instead.) The browser_finder should be refactored so that we are only looking for browsers on the platform specified, or the local platform if no devices are specified.

This bug appears to be specific to #1. If it happens again, we should ask infra to look into the credentials/network access on the bots in question.

I'd be happy to help with 2 or 3. I might get to #3 but won't get to #2 this quarter unless it becomes a higher priority. They should each be tracked in separate bugs since the scope of them is larger than this specific issue.

Comment 8 by aiolos@chromium.org, Apr 20 2016

Labels: Infra-Labs
From comment #7:
#2 is covered in https://crbug.com/591173 which is assigned to me.
#3 we should hijack  https://crbug.com/435723  for this work. It is currently unassigned.

Can someone on Infra comment on if there is anything else we can do for #1 here?

Comment 9 by aiolos@chromium.org, Apr 20 2016

Labels: Infra-Troopers
Components: Infra>Platform>Swarming
Labels: -Infra-Swarming
Components: Infra>Labs
Labels: -Infra-Labs
Labels: -Infra-Troopers
Why is this bug in the trooper queue? I don't see what a trooper can do here. Removing the label.
Status: WontFix (was: Assigned)

Sign in to add a comment