New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 650940 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Sep 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocked on:
issue 650713
issue 651114



Sign in to add a comment

Error running release buildbot on a release branch

Project Member Reported by ameyd@google.com, Sep 28 2016

Issue description

I'm trying to run the lakitu-release trybot on R54 branch (command below), and I'm getting the following errors.  Are those expected?  Any ideas how to work around them?


$ bin/cbuildbot --remote -b release-R54-8743.B -g *291197 lakitu-release
Verifying patches...
Submitting tryjob...
Tryjob submitted!
Go to https://uberchromegw.corp.google.com/i/chromiumos.tryserver/waterfall?committer=ameyd@google.com&builder=release to view the status of your job.

Error from cbuildbot stdio (at step #9):
"""
@@@STEP_CURSOR@cbuildbot [lakitu-release]@@@
step returned non-zero exit code: 1
[E2016-09-27T22:14:30.363351-07:00 27809 0 processor.go:245] Failed to process annotation.               {"error":"STEP_CURSOR could not lookup step: no step named \"cbuildbot [lakitu-release]\"", "annotation":"STEP_CURSOR@cbuildbot [lakitu-release]", "stream":"stdout"}
@@@STEP_FAILURE@@@
"""

Link to the build iteration: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/6454
 

Comment 1 by ameyd@google.com, Sep 28 2016

Cc: shuqianz@chromium.org
Also, I see a slightly different error when running a trybot in R53 branch:

$ bin/cbuildbot --remote -g *291056 -b release-R53-8530.B lakitu-release
Verifying patches...
Submitting tryjob...
Tryjob submitted!
Go to https://uberchromegw.corp.google.com/i/chromiumos.tryserver/waterfall?committer=ameyd@google.com&builder=release to view the status of your job.

Error from "steps" stdio:
"""
INFO:cipd:CIPD client [/b/build/slave/release/.recipe_cipd/cipd] ensuring 2 package(s)...
INFO:cipd:Loading package: infra/tools/luci/logdog/butler/linux-amd64 @ git_revision:43e215ba861ba2130ffe74907d4d3c16e2791928
INFO:cipd:Loading package: infra/tools/luci/logdog/annotee/linux-amd64 @ git_revision:43e215ba861ba2130ffe74907d4d3c16e2791928
INFO:annotated_run:Bootstrapping through LogDog: ['/b/build/slave/release/.recipe_cipd/logdog_butler', '-log-level', 'warning', '-project', 'chromeos', '-prefix', 'bb/chromiumos.tryserver/release/6455', '-output', 'logdog,host="services-dot-luci-logdog.appspot.com"', '-service-account-json', '/creds/service_accounts/service-account-luci-logdog-publisher.json', '-output-max-buffer-age', '30s', 'run', '-stdout', 'tee=stdout', '-stderr', 'tee=stderr', '-streamserver-uri', 'unix:/b/build/.recipe_runtime/tmpEgkFzb/butler.sock', '--', '/b/build/slave/release/.recipe_cipd/logdog_annotee', '-log-level', 'warning', '-project', 'chromeos', '-butler-stream-server', 'unix:/b/build/.recipe_runtime/tmpEgkFzb/butler.sock', '-logdog-host', 'luci-logdog.appspot.com', '-annotate', 'tee', '-name-base', 'recipes', '-print-summary', '-tee', '-json-args-path', '/b/build/slave/release/.recipe_runtime/tmpRit6oh/logdog_annotee_cmd.json', '-result-path', '/b/build/slave/release/.recipe_runtime/tmpRit6oh/bootstrap_result.json']
INFO:root:Running ['git', 'rev-parse', '--verify', '69af4e53c1be1b0fbbc9e57edd70926b8d4f9370^{commit}']
fatal: Needed a single revision
"""

Link to the build iteration: https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/6455
Project Member

Comment 2 by sheriffbot@chromium.org, Sep 28 2016

Labels: Hotlist-Google

Comment 3 by ameyd@google.com, Sep 28 2016

Summary: Error running release buildbot on a release branch (was: Error running trybot on a release branch)
Note that I'm also getting a similar error on the lakitu-release buildbot on R53:

"""
INFO:root:Running ['git', 'rev-parse', '--verify', '69af4e53c1be1b0fbbc9e57edd70926b8d4f9370^{commit}']
fatal: Needed a single revision
"""

Full builder log: https://uberchromegw.corp.google.com/i/chromeos_release/builders/lakitu-release%20release-R53-8530.B/builds/88
Owner: nxia@chromium.org
ningning, can you take a look?

Comment 5 by ameyd@google.com, Sep 28 2016

Labels: -Pri-1 Pri-0
Raising to P0 because we need to push a new 53 stable version today ... It will be very helpful to have a workaround to build _a_ version asap while the builder gets fixed.  Thoughts?

Comment 6 by ameyd@google.com, Sep 28 2016

Labels: -Pri-0 Pri-1
Forcing a "clobbered" build seems to have helped get the release builder past the initial failure: https://uberchromegw.corp.google.com/i/chromeos_release/builders/lakitu-release%20release-R53-8530.B/builds/89

I'll keep this bug open until we get some eyes on the trybots issue.

Comment 7 by nxia@chromium.org, Sep 28 2016

the cleanup failure is recorded at  crbug.com/651114 .

the --git-cache-dir failure for old release branch is tracked at  crbug.com/650713 . dgarrett@ has a CL to fix this from cbuildbot:https://chromium-review.googlesource.com/#/c/389873/. dnj@ should be fixing it from chrome_infra.

Comment 8 by nxia@chromium.org, Sep 28 2016

Blockedon: 650713 651114
comment#1 is about a tryjob "$ bin/cbuildbot --remote -b release-R54-8743.B -g *291197 lakitu-release", which is blocked by two issues mentioned above.

but comment#6 runs a real prod builder with clobber, so it bypasses two issues

"bin/cbuildbot --buildroot /b/cbuild/shared_internal --buildbot --branch release-R53-8530.B --buildnumber 89 --clobber --repo-cache /var/cache/chrome-infra/ccompute-setup/cros-internal lakitu-release",

Please note don't clobber too many builders to bypass the cleanup error, we will run out of GoB quota. People should have been working on merge the cleanup fix to the old release branches.



Comment 9 by ameyd@google.com, Sep 28 2016

Acknowledged, thanks for clarifying!  This was probably the first time I have clicked "force build" with "clobber" on a release builder in a year or so, because we needed an official build on R53 somewhat urgently.  I think we can wait for the correct fix for other builders and trybots.
Also, "bin/cbuildbot --remote --buildbot" isn't a supported combination. As you have found previously, it doesn't do everything a real release build does.

We mangle the build configs of all trybots before running them to avoid conflicts with real production builders.

Comment 11 by nxia@chromium.org, Sep 29 2016

Status: Fixed (was: Untriaged)
dnj@ has fixed this issue.

Sign in to add a comment