New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 615467 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Closed: Sep 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

Exception in WaitUntilReady is not recorded properly.

Project Member Reported by dgarr...@chromium.org, May 27 2016

Issue description

If an exception is raised during WaitUntilReady(), the log will be recorded in a fairly free form manner.

It is probably better if we catch the exception in GenericStage.Run(), and split out the stage logs as we normally do for stages so that the stage will appear on the waterfall, and be marked red.
 
Labels: -current-issue
Owner: nxia@chromium.org
Status: Available (was: Untriaged)
Labels: BuildHealth iptaskforce
Owner: pprabhu@chromium.org
Status: Started (was: Available)
For example, I inserted an exception in the CleanupStage, and obtained the following log:

@@@BUILD_STEP@BuildStart@@@
************************************************************
** Start Stage BuildStart - Thu, 15 Sep 2016 15:39:37 -0700 (PDT)
** 
** The first stage to run.
** 
**   This stage writes a few basic metadata values that are known at the start of
**   build, and inserts the build into the database, if appropriate.
************************************************************

@@@STEP_LINK@Builder documentation@http://www.chromium.org/chromium-os/build/builder-overview#TOC-Pre-CQ@@@
************************************************************
** Finished Stage BuildStart - Thu, 15 Sep 2016 15:39:37 -0700 (PDT)
************************************************************
Traceback (most recent call last):
  File "/work/chromiumos/chromite/cbuildbot/builders/generic_builders.py", line 296, in Run
    self.Initialize()
  File "/work/chromiumos/chromite/cbuildbot/builders/generic_builders.py", line 64, in Initialize
    self._RunStage(build_stages.CleanUpStage)
  File "/work/chromiumos/chromite/cbuildbot/builders/generic_builders.py", line 107, in _RunStage
    return stage_instance.Run()
  File "/work/chromiumos/chromite/cbuildbot/stages/generic_stages.py", line 496, in Run
    ready = self.WaitUntilReady()
  File "/work/chromiumos/chromite/cbuildbot/stages/build_stages.py", line 57, in WaitUntilReady
    raise Exception('[xkcd] Die. Die. Why won\'t you die?')
Exception: [xkcd] Die. Die. Why won't you die?

@@@BUILD_STEP@Report@@@
************************************************************
** Start Stage Report - Thu, 15 Sep 2016 15:39:37 -0700 (PDT)
** 
** Summarize all the builds.

[...snip...]

Note that the exception shows up in a stage prior to the actual stage, and that the relevant stage isn't even started.

After the fix, the output for the same failure is as expected:

pprabhu@pprabhu:/work/chromiumos/chromite$ cbuildbot --local -r /cache_ssd/trybot binhost-pre-cq
15:41:47: INFO: cbuildbot was executed with args /work/chromiumos/chromite/bin/cbuildbot --local -r /cache_ssd/trybot binhost-pre-cq

@@@BUILD_STEP@BuildStart@@@
************************************************************
** Start Stage BuildStart - Thu, 15 Sep 2016 15:41:47 -0700 (PDT)
** 
** The first stage to run.
** 
**   This stage writes a few basic metadata values that are known at the start of
**   build, and inserts the build into the database, if appropriate.
************************************************************

@@@STEP_LINK@Builder documentation@http://www.chromium.org/chromium-os/build/builder-overview#TOC-Pre-CQ@@@
************************************************************
** Finished Stage BuildStart - Thu, 15 Sep 2016 15:41:47 -0700 (PDT)
************************************************************

@@@BUILD_STEP@CleanUp@@@
************************************************************
** Start Stage CleanUp - Thu, 15 Sep 2016 15:41:47 -0700 (PDT)
** 
** Stages that cleans up build artifacts from previous runs.
** 
**   This stage cleans up previous KVM state, temporary git commits,
**   clobbers, and wipes tmp inside the chroot.
************************************************************
Traceback (most recent call last):
  File "/work/chromiumos/chromite/cbuildbot/builders/generic_builders.py", line 296, in Run
    self.Initialize()
  File "/work/chromiumos/chromite/cbuildbot/builders/generic_builders.py", line 64, in Initialize
    self._RunStage(build_stages.CleanUpStage)
  File "/work/chromiumos/chromite/cbuildbot/builders/generic_builders.py", line 107, in _RunStage
    return stage_instance.Run()
  File "/work/chromiumos/chromite/cbuildbot/stages/generic_stages.py", line 500, in Run
    ready = self.WaitUntilReady()
  File "/work/chromiumos/chromite/cbuildbot/stages/build_stages.py", line 57, in WaitUntilReady
    raise Exception('[xkcd] Die. Die. Why won\'t you die?')
Exception: [xkcd] Die. Die. Why won't you die?

@@@BUILD_STEP@Report@@@
************************************************************
** Start Stage Report - Thu, 15 Sep 2016 15:41:47 -0700 (PDT)

[...snip...]
Project Member

Comment 6 by bugdroid1@chromium.org, Sep 26 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/95d596b17c7ee02a9b65a3dbbcf5ad296da97c60

commit 95d596b17c7ee02a9b65a3dbbcf5ad296da97c60
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Thu Sep 15 22:39:01 2016

cbuildbot: Handle stage failures while skipping / waiting uniformly.

GenericStages.WaitUntilReady is a mechanism for cbuildbot stages to
specify preconditions that the stage waits for, but that should not
strictly be considered part of the stage. GenericStages.HandleSkip is a
mechanism to skip a stage.
This CL ensures that failures in these functions are handled the same
way as failures when actually performing a stage.

BUG= chromium:615467 
BUG= chromium:644436 
BUG= chromium:425249 
TEST=(1) cbuildbot/run_tests
     (2) Inject failure in a stage and verify that exception shows up in
         the logs for that stage.
     (3) Inject failures in HandleSkip for a stage and verify that
         exception shows up in the logs for that stage.

Change-Id: I2aef9a375f3144f8261cbad0a4452e74ce63bfd6
Reviewed-on: https://chromium-review.googlesource.com/386141
Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org>
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/95d596b17c7ee02a9b65a3dbbcf5ad296da97c60/cbuildbot/stages/generic_stages.py

Status: Fixed (was: Started)

Comment 8 by dchan@chromium.org, Oct 7 2016

Labels: VerifyIn-55

Comment 9 by dchan@chromium.org, Oct 10 2016

Labels: -VerifyIn-55

Comment 10 by dchan@google.com, Nov 19 2016

Labels: VerifyIn-56

Comment 11 by dchan@google.com, Jan 21 2017

Labels: VerifyIn-57

Comment 12 by dchan@google.com, Mar 4 2017

Labels: VerifyIn-58

Comment 13 by dchan@google.com, Apr 17 2017

Labels: VerifyIn-59

Comment 14 by dchan@google.com, May 30 2017

Labels: VerifyIn-60
Labels: VerifyIn-61

Comment 16 by dchan@chromium.org, Oct 14 2017

Status: Archived (was: Fixed)

Sign in to add a comment