Logs from WaitUntilReady are not recorded properly. |
|||||||||||||
Issue description
Recently, I made a change to add a WaitUntilReady method for the HWTest stages and move the appropriate logic in there. As a side effect, the log output now differs and can be confusing.
Previously, the stdio of the HWTest stage would look like this:
@@@BUILD_STEP@HWTest [sanity]@@@
************************************************************
** Start Stage HWTest [sanity] - Mon, 29 Aug 2016 12:48:23 -0700 (PDT)
**
** Stage that runs tests in the Autotest lab.
************************************************************
12:48:23: INFO: Waiting up to forever for payloads ...
12:53:02: INFO: RunCommand: /b/cbuild/internal_master/chromite/third_party/swarming.client/swarming.py run
[..snip..]
But now, it looks like this:
@@@BUILD_STEP@HWTest [sanity]@@@
************************************************************
** Start Stage HWTest [sanity] - Mon, 05 Sep 2016 20:49:08 -0700 (PDT)
**
** Stage that runs tests in the Autotest lab.
************************************************************
20:49:08: INFO: RunCommand: /b/cbuild/internal_master/chromite/third_party/swarming.client/swarming.py run
[..snip..]
21:07:25: INFO: cidb query succeeded after 1 retries
21:07:25: INFO: Running cidb query on pid 12642, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f071605b810>
************************************************************
** Finished Stage HWTest [sanity] - Mon, 05 Sep 2016 21:07:25 -0700 (PDT)
************************************************************
21:07:25: INFO: Running cidb query on pid 11985, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7f0716048e50>
21:07:25: WARNING: <class 'sqlalchemy.exc.OperationalError'>((OperationalError) (2006, 'MySQL server has gone away') 'INSERT INTO `buildStageTable` (build_id, name, board, status) VALUES (%s, %s, %s, %s)' (1045094, 'HWTest [bvt-inline]', 'auron_yuna', 'planned'))
21:07:29: INFO: cidb query succeeded after 1 retries
21:07:29: INFO: Running cidb query on pid 11985, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7f0716050710>
21:07:29: INFO: Created cidb engine bot@173.194.81.53 for pid 18623
21:07:29: INFO: Running cidb query on pid 18623, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f07163627d0>
21:07:30: INFO: Waiting up to forever for payloads and test artifacts ...
21:07:30: INFO: Running cidb query on pid 18623, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f0716048950>
Basically, the info log message about "Waiting up to forever for payloads" is now printed after the end of the stage.
And actually, I think that log message belongs to the *next* stage because of the timestamps. The suite shouldn't have been created until after WaitUntilReady() returned True. Examining the steps stdio shows that.
21:02:36: INFO: cidb query succeeded after 1 retries
************************************************************
** Finished Stage Archive - Mon, 05 Sep 2016 21:02:36 -0700 (PDT)
************************************************************
20:41:27: INFO: Created cidb engine bot@173.194.81.53 for pid 11985
20:41:27: INFO: Running cidb query on pid 11985, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7f0714f29ed0>
20:41:27: INFO: Created cidb engine bot@173.194.81.53 for pid 12642
20:41:27: INFO: Running cidb query on pid 12642, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f07163627d0>
20:41:28: INFO: Waiting up to forever for payloads and test artifacts ...
20:49:03: INFO: Running cidb query on pid 12642, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f071603fa50>
20:49:03: WARNING: <class 'sqlalchemy.exc.OperationalError'>((OperationalError) (2006, 'MySQL server has gone away') 'UPDATE `buildStageTable` SET status=%s, start_time=CURRENT_TIMESTAMP WHERE `buildStageTable`.id = %s' ('inflight', 29046774L))
20:49:08: INFO: cidb query succeeded after 1 retries
@@@BUILD_STEP@HWTest [sanity]@@@
************************************************************
** Start Stage HWTest [sanity] - Mon, 05 Sep 2016 20:49:08 -0700 (PDT)
[..snip..]
,
Sep 14 2016
,
Sep 15 2016
,
Sep 26 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/95d596b17c7ee02a9b65a3dbbcf5ad296da97c60 commit 95d596b17c7ee02a9b65a3dbbcf5ad296da97c60 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Thu Sep 15 22:39:01 2016 cbuildbot: Handle stage failures while skipping / waiting uniformly. GenericStages.WaitUntilReady is a mechanism for cbuildbot stages to specify preconditions that the stage waits for, but that should not strictly be considered part of the stage. GenericStages.HandleSkip is a mechanism to skip a stage. This CL ensures that failures in these functions are handled the same way as failures when actually performing a stage. BUG= chromium:615467 BUG= chromium:644436 BUG= chromium:425249 TEST=(1) cbuildbot/run_tests (2) Inject failure in a stage and verify that exception shows up in the logs for that stage. (3) Inject failures in HandleSkip for a stage and verify that exception shows up in the logs for that stage. Change-Id: I2aef9a375f3144f8261cbad0a4452e74ce63bfd6 Reviewed-on: https://chromium-review.googlesource.com/386141 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> [modify] https://crrev.com/95d596b17c7ee02a9b65a3dbbcf5ad296da97c60/cbuildbot/stages/generic_stages.py
,
Sep 26 2016
,
Oct 7 2016
,
Oct 10 2016
,
Nov 19 2016
,
Jan 21 2017
,
Mar 4 2017
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Oct 14 2017
|
|||||||||||||
►
Sign in to add a comment |
|||||||||||||
Comment 1 by dgarr...@chromium.org
, Sep 8 2016