pre-cq does not protect us from failures in ReportStage |
|||||||||||
Issue descriptionThe change https://chromium-review.googlesource.com/#/c/366803/ resulted in multiple failing builds, but the Pre-Commit Queue cleared the build to go through to the CQ. https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/compile-only-pre-cq/builds/53490 https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/compile-only-pre-cq/builds/53491 https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/compile-only-pre-cq/builds/53492 Perhaps failures of the Report stage don't properly get treated as build failures?
,
Aug 13 2016
,
Aug 16 2016
Aviv, can you take a look?
,
Aug 17 2016
I don't have build logs anymore. That makes this pretty hard to debug. Let's see if it happens again.
,
Aug 29 2016
Re OP ah, yes, ReportStage failures will not get treated as failures by the pre-cq. In the current architecture, pre-cq status is recorded by the slave in PreCQCompletion stage. That runs before ReportStage. Whereas in the CQ, the master examines the final status of each slave build, which is recorded in ReportStage. This would take some design work to fix. I'm not sure how much. Let's leave this open and see when it next hits us. Maybe we can mitigate. Alternative would be to add logic to the pre-cq-launcher so that it looks at status of builds as well, but that is adding complexity to an already complex piece of machinery.
,
Nov 3 2016
Perhaps nxia@ can take a look, though this is not the higest priority.
,
Feb 3 2017
When a pre-cq passes the completion stage, it inserts CL_ACTION_VERIFIED into CIDB to mark this CL as verified. One option here is to query the pre-cq status in Buildbucket. If the build fails at ReportStage, its result will be 'FAILURE' in Buildbucket. The question here is do we care about ReportStages for all pre-cqs, or only some specific pre-cqs.
,
Feb 6 2017
Re c#7: if the pre-cq run is testing a chromite change, then yes.
,
Feb 7 2017
The change will be fairly big, I'm not sure if it's worthy to make the change only for chromite. I usually use tryjobs to test chromite changes. And chromite sync change can never be tested by Pre-CQ so I just use unittest to test sync changes. akeshet@ may comment here.
,
Mar 9 2017
The title of this bug is correct. However, it isn't so simple to fix. Maybe a fixit candidate, if somebody can think of a simple solution.
,
Mar 15 2017
,
Dec 13 2017
,
Jun 8 2018
|
|||||||||||
►
Sign in to add a comment |
|||||||||||
Comment 1 by tandrii@chromium.org
, Aug 10 2016