coral-paladin seems not to matter
Reported by
jrbarnette@chromium.org,
Jan 4 2018
|
|||||||
Issue description
The coral-paladin builder was marked non-experimental for a long stretch
over the Christmas break. During that stretch, the builder never passed.
Despite all the failures, the CQ _did_ pass, apparently ignoring the
helpless flailing of a builder that it ought to have cared about.
For instance, this build failed:
https://uberchromegw.corp.google.com/i/chromeos/builders/coral-paladin/builds/2193
The corresponding master build passed with flying colors:
https://luci-milo.appspot.com/buildbot/chromeos/master-paladin/17385
The CommitQueueCompletion log is here:
https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fmaster-paladin%2F17385%2F%2B%2Frecipes%2Fsteps%2FCommitQueueCompletion%2F0%2Fstdout
Digging into ththat log, I note this line:
06:14:32: INFO: Waiting for the following builds to complete: [ ... omitted due to length ...]
That line doesn't mention "coral-paladin" in the list of builders to
wait for. In fact, coral-paladin is mentioned nowhere in the log.
So, basically, whatever normally tells the CQ which slave paladins
matter has overlooked the coral-paladin slave.
,
Jan 4 2018
In the linked build, coral-paladin is important=false
,
Jan 5 2018
It is also failing, AND marked experimental in build status. https://uberchromegw.corp.google.com/i/chromeos/builders/coral-paladin If it's expected to be green, but dealing with a temporary failure, it should be important in the config, and experimental in the build status. If it's still a new untrusted builder, the current config is correct, but the build status should be cleared, and there should be someone owning the process of making it green.
,
Jan 5 2018
,
Jan 5 2018
I think Charles might know what the team is doing there. I'm not sure.
,
Jan 5 2018
,
Jan 5 2018
+YH whom may know about investigations into Coral hwtest stability.
,
Jan 5 2018
There have been ongoing infrastructure stability issues. The most recent one was the alloc for astronaut was taken away for some reason, but that has since been resolved. It appears to be another issue now. Either way, I've yet to see something that wasn't infrastructure related to these flakes yet.
,
Jan 5 2018
So.... you are saying we should mark the build as important in both the build config, and the build status?
,
Jan 5 2018
it was unmarked base on crbug.com/788584 I would defer to that as the main root of determining whether to mark as important or not
,
Jan 5 2018
I just started to make that change, and found that it's marked as experimental because of https://crbug.com/788584. Is that still relevant?
,
Jan 5 2018
,
Jan 6 2018
It's not clear to me whether bug 788584 is still a thing or not. However, I think our best way forward is this: 1) Commit the change to make the builder important again. 2) Leave the builder EXPERIMENTAL in the tree status. 3) Work to resolve the problems and make the builder green. Regarding item 3), we'll probably need to file new bugs for whatever problems there are, and we'll need engagement from sheriffs and/or coral experts to debug the problems.
,
Jan 6 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/0e1d4133f9c0ef5a5cffb04591ee6d939a66b074 commit 0e1d4133f9c0ef5a5cffb04591ee6d939a66b074 Author: Don Garrett <dgarrett@google.com> Date: Sat Jan 06 05:14:14 2018 chromeos_config: Mark coral-paladin important. BUG= chromium:799196 ,chromium:788584 TEST=chromeos_config_unittest Change-Id: I747a4310f7753459f42ed47270dd06fa97be74e5 Reviewed-on: https://chromium-review.googlesource.com/853114 Commit-Ready: Don Garrett <dgarrett@chromium.org> Tested-by: Don Garrett <dgarrett@chromium.org> Reviewed-by: Richard Barnette <jrbarnette@google.com> Reviewed-by: C Shapiro <shapiroc@google.com> [modify] https://crrev.com/0e1d4133f9c0ef5a5cffb04591ee6d939a66b074/cbuildbot/config_dump.json [modify] https://crrev.com/0e1d4133f9c0ef5a5cffb04591ee6d939a66b074/cbuildbot/chromeos_config.py
,
Jan 8 2018
I'm calling this fixed, since the builder is now important. Regarding making the paladin green, there's bug 799997 ; I'm reliably informed that that's our only known obstacle. |
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by jrbarnette@chromium.org
, Jan 4 2018Owner: dgarr...@chromium.org
Status: Assigned (was: Untriaged)