Exception in WaitUntilReady is not recorded properly. |
|||||||||||||||
Issue descriptionIf an exception is raised during WaitUntilReady(), the log will be recorded in a fairly free form manner. It is probably better if we catch the exception in GenericStage.Run(), and split out the stage logs as we normally do for stages so that the stage will appear on the waterfall, and be marked red.
,
Jul 29 2016
,
Sep 8 2016
,
Sep 14 2016
,
Sep 15 2016
For example, I inserted an exception in the CleanupStage, and obtained the following log: @@@BUILD_STEP@BuildStart@@@ ************************************************************ ** Start Stage BuildStart - Thu, 15 Sep 2016 15:39:37 -0700 (PDT) ** ** The first stage to run. ** ** This stage writes a few basic metadata values that are known at the start of ** build, and inserts the build into the database, if appropriate. ************************************************************ @@@STEP_LINK@Builder documentation@http://www.chromium.org/chromium-os/build/builder-overview#TOC-Pre-CQ@@@ ************************************************************ ** Finished Stage BuildStart - Thu, 15 Sep 2016 15:39:37 -0700 (PDT) ************************************************************ Traceback (most recent call last): File "/work/chromiumos/chromite/cbuildbot/builders/generic_builders.py", line 296, in Run self.Initialize() File "/work/chromiumos/chromite/cbuildbot/builders/generic_builders.py", line 64, in Initialize self._RunStage(build_stages.CleanUpStage) File "/work/chromiumos/chromite/cbuildbot/builders/generic_builders.py", line 107, in _RunStage return stage_instance.Run() File "/work/chromiumos/chromite/cbuildbot/stages/generic_stages.py", line 496, in Run ready = self.WaitUntilReady() File "/work/chromiumos/chromite/cbuildbot/stages/build_stages.py", line 57, in WaitUntilReady raise Exception('[xkcd] Die. Die. Why won\'t you die?') Exception: [xkcd] Die. Die. Why won't you die? @@@BUILD_STEP@Report@@@ ************************************************************ ** Start Stage Report - Thu, 15 Sep 2016 15:39:37 -0700 (PDT) ** ** Summarize all the builds. [...snip...] Note that the exception shows up in a stage prior to the actual stage, and that the relevant stage isn't even started. After the fix, the output for the same failure is as expected: pprabhu@pprabhu:/work/chromiumos/chromite$ cbuildbot --local -r /cache_ssd/trybot binhost-pre-cq 15:41:47: INFO: cbuildbot was executed with args /work/chromiumos/chromite/bin/cbuildbot --local -r /cache_ssd/trybot binhost-pre-cq @@@BUILD_STEP@BuildStart@@@ ************************************************************ ** Start Stage BuildStart - Thu, 15 Sep 2016 15:41:47 -0700 (PDT) ** ** The first stage to run. ** ** This stage writes a few basic metadata values that are known at the start of ** build, and inserts the build into the database, if appropriate. ************************************************************ @@@STEP_LINK@Builder documentation@http://www.chromium.org/chromium-os/build/builder-overview#TOC-Pre-CQ@@@ ************************************************************ ** Finished Stage BuildStart - Thu, 15 Sep 2016 15:41:47 -0700 (PDT) ************************************************************ @@@BUILD_STEP@CleanUp@@@ ************************************************************ ** Start Stage CleanUp - Thu, 15 Sep 2016 15:41:47 -0700 (PDT) ** ** Stages that cleans up build artifacts from previous runs. ** ** This stage cleans up previous KVM state, temporary git commits, ** clobbers, and wipes tmp inside the chroot. ************************************************************ Traceback (most recent call last): File "/work/chromiumos/chromite/cbuildbot/builders/generic_builders.py", line 296, in Run self.Initialize() File "/work/chromiumos/chromite/cbuildbot/builders/generic_builders.py", line 64, in Initialize self._RunStage(build_stages.CleanUpStage) File "/work/chromiumos/chromite/cbuildbot/builders/generic_builders.py", line 107, in _RunStage return stage_instance.Run() File "/work/chromiumos/chromite/cbuildbot/stages/generic_stages.py", line 500, in Run ready = self.WaitUntilReady() File "/work/chromiumos/chromite/cbuildbot/stages/build_stages.py", line 57, in WaitUntilReady raise Exception('[xkcd] Die. Die. Why won\'t you die?') Exception: [xkcd] Die. Die. Why won't you die? @@@BUILD_STEP@Report@@@ ************************************************************ ** Start Stage Report - Thu, 15 Sep 2016 15:41:47 -0700 (PDT) [...snip...]
,
Sep 26 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/95d596b17c7ee02a9b65a3dbbcf5ad296da97c60 commit 95d596b17c7ee02a9b65a3dbbcf5ad296da97c60 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Thu Sep 15 22:39:01 2016 cbuildbot: Handle stage failures while skipping / waiting uniformly. GenericStages.WaitUntilReady is a mechanism for cbuildbot stages to specify preconditions that the stage waits for, but that should not strictly be considered part of the stage. GenericStages.HandleSkip is a mechanism to skip a stage. This CL ensures that failures in these functions are handled the same way as failures when actually performing a stage. BUG= chromium:615467 BUG= chromium:644436 BUG= chromium:425249 TEST=(1) cbuildbot/run_tests (2) Inject failure in a stage and verify that exception shows up in the logs for that stage. (3) Inject failures in HandleSkip for a stage and verify that exception shows up in the logs for that stage. Change-Id: I2aef9a375f3144f8261cbad0a4452e74ce63bfd6 Reviewed-on: https://chromium-review.googlesource.com/386141 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/95d596b17c7ee02a9b65a3dbbcf5ad296da97c60/cbuildbot/stages/generic_stages.py
,
Sep 26 2016
,
Oct 7 2016
,
Oct 10 2016
,
Nov 19 2016
,
Jan 21 2017
,
Mar 4 2017
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Oct 14 2017
|
|||||||||||||||
►
Sign in to add a comment |
|||||||||||||||
Comment 1 by autumn@chromium.org
, Jun 2 2016