New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 636066 link

Starred by 1 user

Issue metadata

Status: Available
Owner: ----
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 3
Type: Bug



Sign in to add a comment

pre-cq does not protect us from failures in ReportStage

Project Member Reported by davidri...@chromium.org, Aug 9 2016

Issue description

Components: -Infra>CQ Infra>Client>ChromeOS
please don't use infra>cq for chrome os cq.

Comment 2 by dshi@chromium.org, Aug 13 2016

Cc: akes...@chromium.org

Comment 3 by aut...@google.com, Aug 16 2016

Owner: akes...@chromium.org
Aviv, can you take a look? 
Status: Unconfirmed (was: Untriaged)
I don't have build logs anymore. That makes this pretty hard to debug. Let's see if it happens again.
Summary: pre-cq does not protect us from failures in ReportStage (was: Pre-commit queue incorrectly passing failing builds)
Re OP ah, yes, ReportStage failures will not get treated as failures by the pre-cq. In the current architecture, pre-cq status is recorded by the slave in PreCQCompletion stage. That runs before ReportStage. Whereas in the CQ, the master examines the final status of each slave build, which is recorded in ReportStage.

This would take some design work to fix. I'm not sure how much. Let's leave this open and see when it next hits us. Maybe we can mitigate. Alternative would be to add logic to the pre-cq-launcher so that it looks at status of builds as well, but that is adding complexity to an already complex piece of machinery.
Owner: nxia@chromium.org
Perhaps nxia@ can take a look, though this is not the higest priority.

Comment 7 by nxia@chromium.org, Feb 3 2017

When a pre-cq passes the completion stage, it inserts CL_ACTION_VERIFIED into CIDB to mark this CL as verified.
One option here is to query the pre-cq status in Buildbucket. If the build fails at ReportStage, its result will be 'FAILURE' in Buildbucket.

The question here is do we care about ReportStages for all pre-cqs, or only some specific pre-cqs.
Re c#7: if the pre-cq run is testing a chromite change, then yes.

Comment 9 by nxia@chromium.org, Feb 7 2017

Cc: nxia@chromium.org
Owner: akes...@chromium.org
The change will be fairly big, I'm not sure if it's worthy to make the change only for chromite. I usually use tryjobs to test chromite changes. And chromite sync change can never be tested by Pre-CQ so I just use unittest to test sync changes. 
akeshet@ may comment here. 
Owner: ----
Status: Available (was: Unconfirmed)
The title of this bug is correct. However, it isn't so simple to fix. Maybe a fixit candidate, if somebody can think of a simple solution.
Labels: OKR Hotlist-Fixit
Labels: -Pri-2 -OKR Pri-3

Comment 13 by nxia@chromium.org, Jun 8 2018

Cc: -nxia@chromium.org

Sign in to add a comment