New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 719680 link

Starred by 1 user
Status: WontFix
Owner:
█ dbehr@chromium.org
Last visit > 30 days ago
Closed: Jun 2017
Cc:
ayatane@chromium.org
jwer...@chromium.org
█ rbhagavatula@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Chrome LKGM builder failing cros_best_revision due to no daisy-release build available

Project Member Reported by jamescook@chromium.org, May 8 2017 
The problem is that there hasn't been a daisy-release in 5 days.

https://uberchromegw.corp.google.com/i/chromeos/builders/daisy-release

There's nothing wrong with the bot itself or with chrome.

00:02:18: INFO: Considering the following versions: 9531.0.0 9530.0.0 9529.0.0 9528.0.0 9527.0.0 9526.0.0 9525.0.0 9524.0.0 9523.0.0 9522.0.0
00:02:18: INFO: Using scores from the following canaries: lumpy-release daisy-release parrot-release
00:02:18: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9531.0.0/lumpy-release
00:02:19: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9531.0.0/daisy-release
00:02:20: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9531.0.0/parrot-release
00:02:20: WARNING: Version: 9531.0.0 Failed builders: daisy-release
00:02:20: INFO: Version: 9531.0.0 Score: -1
00:02:20: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9530.0.0/lumpy-release
00:02:21: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9530.0.0/daisy-release
00:02:21: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9530.0.0/parrot-release
00:02:22: WARNING: Version: 9530.0.0 Failed builders: daisy-release
00:02:22: INFO: Version: 9530.0.0 Score: -1
00:02:22: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9529.0.0/lumpy-release
00:02:22: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9529.0.0/daisy-release
00:02:23: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9529.0.0/parrot-release
00:02:23: WARNING: Version: 9529.0.0 Failed builders: daisy-release
00:02:23: INFO: Version: 9529.0.0 Score: -1
00:02:23: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9528.0.0/lumpy-release
00:02:24: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9528.0.0/daisy-release
00:02:24: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9528.0.0/parrot-release
00:02:25: WARNING: Version: 9528.0.0 Failed builders: daisy-release
00:02:25: INFO: Version: 9528.0.0 Score: -1
00:02:25: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9527.0.0/lumpy-release
00:02:26: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9527.0.0/daisy-release
00:02:26: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9527.0.0/parrot-release
00:02:27: WARNING: Version: 9527.0.0 Failed builders: daisy-release
00:02:27: INFO: Version: 9527.0.0 Score: -1
00:02:27: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9526.0.0/lumpy-release
00:02:27: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9526.0.0/daisy-release
00:02:28: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9526.0.0/parrot-release
00:02:29: WARNING: Version: 9526.0.0 Failed builders: daisy-release
00:02:29: INFO: Version: 9526.0.0 Score: -1
00:02:29: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9525.0.0/lumpy-release
00:02:29: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9525.0.0/daisy-release
00:02:30: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9525.0.0/parrot-release
00:02:30: WARNING: Version: 9525.0.0 Failed builders: daisy-release
00:02:30: INFO: Version: 9525.0.0 Score: -1
00:02:30: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9524.0.0/lumpy-release
00:02:31: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9524.0.0/daisy-release
00:02:32: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9524.0.0/parrot-release
00:02:32: WARNING: Version: 9524.0.0 Failed builders: daisy-release
00:02:32: INFO: Version: 9524.0.0 Score: -1
00:02:32: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9523.0.0/lumpy-release
00:02:33: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9523.0.0/daisy-release
00:02:34: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9523.0.0/parrot-release
00:02:35: WARNING: Version: 9523.0.0 Failed builders: daisy-release
00:02:35: INFO: Version: 9523.0.0 Score: -1
00:02:35: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9522.0.0/lumpy-release
00:02:35: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9522.0.0/daisy-release
00:02:36: INFO: RunCommand: /b/build/slave/chromelkgm/build/.cros_cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' cat gs://chromeos-manifest-versions/builder-status/9522.0.0/parrot-release
00:02:36: WARNING: Version: 9522.0.0 Failed builders: daisy-release
00:02:36: INFO: Version: 9522.0.0 Score: -1

cros_best_revision: Unhandled exception:
Traceback (most recent call last):
  File "/b/build/slave/chromelkgm/build/chromite/bin/cros_best_revision", line 168, in <module>
    DoMain()
  File "/b/build/slave/chromelkgm/build/chromite/bin/cros_best_revision", line 164, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/b/build/slave/chromelkgm/build/chromite/lib/commandline.py", line 837, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/b/build/slave/chromelkgm/build/chromite/scripts/cros_best_revision.py", line 262, in main
    committer.FindNewLKGM()
  File "/b/build/slave/chromelkgm/build/chromite/scripts/cros_best_revision.py", line 144, in FindNewLKGM
    raise LKGMNotFound('No valid LKGM found. Scores are too low.')
chromite.scripts.cros_best_revision.LKGMNotFound: No valid LKGM found. Scores are too low.
step returned non-zero exit code: 1

Chrome OS sheriffs - is there something in particular wrong with daisy-release? A bug this can be linked against / duped into?

Comment 1 by jwer...@chromium.org, May 8 2017 

Looking at the last failures on daisy release, we had:

- 10 Paygen issues, I assume all related to issue 719322 / issue 716638
- https://uberchromegw.corp.google.com/i/chromeos/builders/daisy-release/builds/1091 is an autoupdate_Rollback error that seems to be also somewhat related (similar error messages as  issue 700033 )
- https://uberchromegw.corp.google.com/i/chromeos/builders/daisy-release/builds/1099 seems to have succeeded everything except that provisioning failed with "Last AU on this DUT failed, The host has wrong cros-version label". No idea what that means but probably also related to the AU tests?
- https://uberchromegw.corp.google.com/i/chromeos/builders/daisy-release/builds/1101 couldn't SSH in or something

...so, yeah, it seems like autoupdate is broken all over the place.

Comment 2 by aut...@google.com, May 9 2017

Owner: dbehr@chromium.org
@dbehr - were you looking into this already?

Comment 3 by jamescook@chromium.org, May 12 2017 

If we're finding good builds now, and the problem is the username in issue  crbug.com/618430  then this could be closed.

It would be nice to do a manual uprev of the version if possible, since we haven't had one for a while.

Comment 4 by steve...@chromium.org, Jun 14 2017

Status: WontFix (was: Untriaged)
►

Sign in to add a comment