New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 644378 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Closed: Nov 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

PFQ PublishUprevChanges step is misleading for failed runs

Project Member Reported by michae...@chromium.org, Sep 6 2016

Issue description

Context: A master PFQ build could end in several ways, including but not limited to:
1. master ran successfully, ran PublishUprevChanges, and uprev'd successfully
2. master ran successfully, but didn't actually have chrome revisions (takes about 30 minutes for slaves to report back), and didn't run PublishUprevChanges
3. master failed MasterSlaveSyncCompletion, ran PublishUprevChanges, but did not uprev because a slave failed

In the third case, the output from PublishUprevChanges and cros_mark_as_stable makes it seem like the uprev happened anyway.

For example, master-chromium-pfq build 3308 failed due to other builder failures[1]. So, as intended, it did not actually uprev, according to GoldenEye[2] (see chrome "55.0.2843.0", chrome os "8751.0.0-rc1", started "08-28 20:22").

However, the output from PublishUprevChanges is virtually indistinguishable from a build that *does* successfully uprev for realsies.

[1] https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/3308
[2] https://cros-goldeneye.corp.google.com/chromeos/console/listPfqBuild#/ (https://cros-goldeneye.corp.google.com/chromeos/console/listPfqBuild#/

Snippet from PublishUprevChanges step:

> peach_pit-LATEST_RELEASE_CHROME_BINHOST.conf: updating LATEST_RELEASE_CHROME_BINHOST
> 
> Change-Id: I0b6044d22b4c94b808c717606ca452369cb0fd52
> 
> 01:00:10: INFO: Successfully pushed /b/cbuild/internal_master/src/private-overlays/chromeos-partner-overlay to cros-internal merge_branch:refs/pfq/staging_pfq_branch_1031067
> 01:00:21: INFO: For /b/cbuild/internal_master/src/third_party/chromiumos-overlay, using description Marking set of ebuilds as stable
> 
> x86-generic-LATEST_RELEASE_CHROME_BINHOST.conf: updating LATEST_RELEASE_CHROME_BINHOST
> 
> Change-Id: Ia88b05bf895a43cf2852fe98fd30cadb5b712093
...
> Change-Id: Ifdd2895fc3e98986a02995d318d58ecdfcfbec03
> 
> Marking latest_release for chromium-source ebuild with version 55.0.2843.0 as stable.
 
Apparently unsuccessful run pushes changes to staging branch (staging_pfq_branch_1031067) instead of master.

I suggest adding extra log here:
https://chromium-review.googlesource.com/#/c/382396/
Cc: -levarum@chromium.org achuith@chromium.org michae...@chromium.org
Owner: levarum@chromium.org
Project Member

Comment 3 by bugdroid1@chromium.org, Sep 9 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/25352ba86a350a1fc58bfc3f6a9857eca5d5e45b

commit 25352ba86a350a1fc58bfc3f6a9857eca5d5e45b
Author: Lev Rumyantsev <levarum@google.com>
Date: Wed Sep 07 22:53:58 2016

Add extra log for unsuccessful chrome-pfq

BUG= chromium:644378 
TEST=None

Change-Id: I2366834c85fd47e242e490c24484a0418f181d4d
Reviewed-on: https://chromium-review.googlesource.com/382396
Commit-Ready: Lev Rumyantsev <levarum@chromium.org>
Tested-by: Lev Rumyantsev <levarum@chromium.org>
Reviewed-by: Achuith Bhandarkar <achuith@chromium.org>
Reviewed-by: Michael Giuffrida <michaelpg@chromium.org>

[modify] https://crrev.com/25352ba86a350a1fc58bfc3f6a9857eca5d5e45b/scripts/cros_mark_as_stable.py

Owner: michae...@chromium.org
The latest fail log looks like this:

19:03:56: INFO: RunCommand: /b/cbuild/internal_master/chromite/bin/cros_mark_as_stable <...options...>
19:04:17: INFO: PFQ FAILED. Pushing uprev change to staging branch refs/pfq/staging_pfq_branch_1052720
19:04:17: INFO: For /b/cbuild/internal_master/src/private-overlays/chromeos-partner-overlay, using description Marking set of ebuilds as stable
<...change logs...>
19:04:21: INFO: Successfully pushed /b/cbuild/internal_master/src/private-overlays/chromeos-partner-overlay to cros-internal merge_branch:refs/pfq/staging_pfq_branch_1052720
19:04:31: INFO: PFQ FAILED. Pushing uprev change to staging branch refs/pfq/staging_pfq_branch_1052720
19:04:31: INFO: For /b/cbuild/internal_master/src/third_party/chromiumos-overlay, using description Marking set of ebuilds as stable
<...change logs...>
19:04:36: INFO: Successfully pushed /b/cbuild/internal_master/src/third_party/chromiumos-overlay to cros merge_branch:refs/pfq/staging_pfq_branch_1052720

Link to log:

https://uberchromegw.corp.google.com/i/chromeos/builders/master-chromium-pfq/builds/3355/steps/PublishUprevChanges/logs/stdio


Michael, it is now clear enough?
Cc: levarum@chromium.org
Status: Fixed (was: Assigned)

Comment 7 by dchan@google.com, Mar 4 2017

Labels: VerifyIn-58

Comment 8 by dchan@google.com, Apr 17 2017

Labels: VerifyIn-59

Comment 9 by dchan@google.com, May 30 2017

Labels: VerifyIn-60
Labels: VerifyIn-61

Comment 11 by dchan@chromium.org, Oct 14 2017

Status: Archived (was: Fixed)

Sign in to add a comment