New issue
Advanced search Search tips

Issue 799196 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jan 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

coral-paladin seems not to matter

Reported by jrbarnette@chromium.org, Jan 4 2018

Issue description

The coral-paladin builder was marked non-experimental for a long stretch
over the Christmas break.  During that stretch, the builder never passed.
Despite all the failures, the CQ _did_ pass, apparently ignoring the
helpless flailing of a builder that it ought to have cared about.

For instance, this build failed:
    https://uberchromegw.corp.google.com/i/chromeos/builders/coral-paladin/builds/2193

The corresponding master build passed with flying colors:
    https://luci-milo.appspot.com/buildbot/chromeos/master-paladin/17385

The CommitQueueCompletion log is here:
    https://logs.chromium.org/v/?s=chromeos%2Fbb%2Fchromeos%2Fmaster-paladin%2F17385%2F%2B%2Frecipes%2Fsteps%2FCommitQueueCompletion%2F0%2Fstdout

Digging into ththat log, I note this line:
06:14:32: INFO: Waiting for the following builds to complete: [ ... omitted due to length ...]

That line doesn't mention "coral-paladin" in the list of builders to
wait for.  In fact, coral-paladin is mentioned nowhere in the log.
So, basically, whatever normally tells the CQ which slave paladins
matter has overlooked the coral-paladin slave.

 
Cc: shapiroc@chromium.org ayatane@chromium.org
Owner: dgarr...@chromium.org
Status: Assigned (was: Untriaged)
Assigning to dgarrett@ on the theory that, if the problem is
easy to understand, he's the most likely to see it.

In the linked build, coral-paladin is important=false
It is also failing, AND marked experimental in build status.

https://uberchromegw.corp.google.com/i/chromeos/builders/coral-paladin

If it's expected to be green, but dealing with a temporary failure, it should be important in the config, and experimental in the build status.

If it's still a new untrusted builder, the current config is correct, but the build status should be cleared, and there should be someone owning the process of making it green.
Cc: sjg@chromium.org

Comment 5 by sjg@google.com, Jan 5 2018

I think Charles might know what the team is doing there. I'm not sure.
Owner: shapiroc@chromium.org
Cc: yueherngl@chromium.org
+YH whom may know about investigations into Coral hwtest stability.
Owner: dgarr...@chromium.org
There have been ongoing infrastructure stability issues.  The most recent one was the alloc for astronaut was taken away for some reason, but that has since been resolved. 

It appears to be another issue now.

Either way, I've yet to see something that wasn't infrastructure related to these flakes yet.
So.... you are saying we should mark the build as important in both the build config, and the build status?
it was unmarked base on crbug.com/788584

I would defer to that as the main root of determining whether to mark as important or not


I just started to make that change, and found that it's marked as experimental because of https://crbug.com/788584. Is that still relevant?


Cc: grundler@chromium.org
It's not clear to me whether bug 788584 is still a thing or
not.  However, I think our best way forward is this:
 1) Commit the change to make the builder important again.
 2) Leave the builder EXPERIMENTAL in the tree status.
 3) Work to resolve the problems and make the builder green.

Regarding item 3), we'll probably need to file new bugs for whatever
problems there are, and we'll need engagement from sheriffs and/or
coral experts to debug the problems.
Project Member

Comment 14 by bugdroid1@chromium.org, Jan 6 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/0e1d4133f9c0ef5a5cffb04591ee6d939a66b074

commit 0e1d4133f9c0ef5a5cffb04591ee6d939a66b074
Author: Don Garrett <dgarrett@google.com>
Date: Sat Jan 06 05:14:14 2018

chromeos_config: Mark coral-paladin important.

BUG= chromium:799196 ,chromium:788584
TEST=chromeos_config_unittest

Change-Id: I747a4310f7753459f42ed47270dd06fa97be74e5
Reviewed-on: https://chromium-review.googlesource.com/853114
Commit-Ready: Don Garrett <dgarrett@chromium.org>
Tested-by: Don Garrett <dgarrett@chromium.org>
Reviewed-by: Richard Barnette <jrbarnette@google.com>
Reviewed-by: C Shapiro <shapiroc@google.com>

[modify] https://crrev.com/0e1d4133f9c0ef5a5cffb04591ee6d939a66b074/cbuildbot/config_dump.json
[modify] https://crrev.com/0e1d4133f9c0ef5a5cffb04591ee6d939a66b074/cbuildbot/chromeos_config.py

Status: Fixed (was: Assigned)
I'm calling this fixed, since the builder is now important.

Regarding making the paladin green, there's  bug 799997 ; I'm
reliably informed that that's our only known obstacle.

Sign in to add a comment