New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 644436 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Closed: Sep 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

Logs from WaitUntilReady are not recorded properly.

Project Member Reported by aaboagye@chromium.org, Sep 6 2016

Issue description

Recently, I made a change to add a WaitUntilReady method for the HWTest stages and move the appropriate logic in there. As a side effect, the log output now differs and can be confusing.

Previously, the stdio of the HWTest stage would look like this:

@@@BUILD_STEP@HWTest [sanity]@@@
************************************************************
** Start Stage HWTest [sanity] - Mon, 29 Aug 2016 12:48:23 -0700 (PDT)
** 
** Stage that runs tests in the Autotest lab.
************************************************************
12:48:23: INFO: Waiting up to forever for payloads ...
12:53:02: INFO: RunCommand: /b/cbuild/internal_master/chromite/third_party/swarming.client/swarming.py run
[..snip..]


But now, it looks like this:

@@@BUILD_STEP@HWTest [sanity]@@@
************************************************************
** Start Stage HWTest [sanity] - Mon, 05 Sep 2016 20:49:08 -0700 (PDT)
** 
** Stage that runs tests in the Autotest lab.
************************************************************
20:49:08: INFO: RunCommand: /b/cbuild/internal_master/chromite/third_party/swarming.client/swarming.py run
[..snip..]
21:07:25: INFO: cidb query succeeded after 1 retries
21:07:25: INFO: Running cidb query on pid 12642, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f071605b810>
************************************************************
** Finished Stage HWTest [sanity] - Mon, 05 Sep 2016 21:07:25 -0700 (PDT)
************************************************************
21:07:25: INFO: Running cidb query on pid 11985, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7f0716048e50>
21:07:25: WARNING: <class 'sqlalchemy.exc.OperationalError'>((OperationalError) (2006, 'MySQL server has gone away') 'INSERT INTO `buildStageTable` (build_id, name, board, status) VALUES (%s, %s, %s, %s)' (1045094, 'HWTest [bvt-inline]', 'auron_yuna', 'planned'))
21:07:29: INFO: cidb query succeeded after 1 retries
21:07:29: INFO: Running cidb query on pid 11985, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7f0716050710>
21:07:29: INFO: Created cidb engine bot@173.194.81.53 for pid 18623
21:07:29: INFO: Running cidb query on pid 18623, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f07163627d0>
21:07:30: INFO: Waiting up to forever for payloads and test artifacts ...
21:07:30: INFO: Running cidb query on pid 18623, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f0716048950>


Basically, the info log message about "Waiting up to forever for payloads" is now printed after the end of the stage.

And actually, I think that log message belongs to the *next* stage because of the timestamps. The suite shouldn't have been created until after WaitUntilReady() returned True. Examining the steps stdio shows that.

21:02:36: INFO: cidb query succeeded after 1 retries
************************************************************
** Finished Stage Archive - Mon, 05 Sep 2016 21:02:36 -0700 (PDT)
************************************************************
20:41:27: INFO: Created cidb engine bot@173.194.81.53 for pid 11985
20:41:27: INFO: Running cidb query on pid 11985, repr(query) starts with <sqlalchemy.sql.expression.Insert object at 0x7f0714f29ed0>
20:41:27: INFO: Created cidb engine bot@173.194.81.53 for pid 12642
20:41:27: INFO: Running cidb query on pid 12642, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f07163627d0>
20:41:28: INFO: Waiting up to forever for payloads and test artifacts ...
20:49:03: INFO: Running cidb query on pid 12642, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f071603fa50>
20:49:03: WARNING: <class 'sqlalchemy.exc.OperationalError'>((OperationalError) (2006, 'MySQL server has gone away') 'UPDATE `buildStageTable` SET status=%s, start_time=CURRENT_TIMESTAMP WHERE `buildStageTable`.id = %s' ('inflight', 29046774L))
20:49:08: INFO: cidb query succeeded after 1 retries

@@@BUILD_STEP@HWTest [sanity]@@@
************************************************************
** Start Stage HWTest [sanity] - Mon, 05 Sep 2016 20:49:08 -0700 (PDT) 
[..snip..]
 
Labels: BuildHealth iptaskforce
Owner: pprabhu@chromium.org
Status: Started (was: Available)
Project Member

Comment 4 by bugdroid1@chromium.org, Sep 26 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/95d596b17c7ee02a9b65a3dbbcf5ad296da97c60

commit 95d596b17c7ee02a9b65a3dbbcf5ad296da97c60
Author: Prathmesh Prabhu <pprabhu@chromium.org>
Date: Thu Sep 15 22:39:01 2016

cbuildbot: Handle stage failures while skipping / waiting uniformly.

GenericStages.WaitUntilReady is a mechanism for cbuildbot stages to
specify preconditions that the stage waits for, but that should not
strictly be considered part of the stage. GenericStages.HandleSkip is a
mechanism to skip a stage.
This CL ensures that failures in these functions are handled the same
way as failures when actually performing a stage.

BUG= chromium:615467 
BUG= chromium:644436 
BUG= chromium:425249 
TEST=(1) cbuildbot/run_tests
     (2) Inject failure in a stage and verify that exception shows up in
         the logs for that stage.
     (3) Inject failures in HandleSkip for a stage and verify that
         exception shows up in the logs for that stage.

Change-Id: I2aef9a375f3144f8261cbad0a4452e74ce63bfd6
Reviewed-on: https://chromium-review.googlesource.com/386141
Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org>
Tested-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>
Reviewed-by: Don Garrett <dgarrett@chromium.org>

[modify] https://crrev.com/95d596b17c7ee02a9b65a3dbbcf5ad296da97c60/cbuildbot/stages/generic_stages.py

Status: Fixed (was: Started)

Comment 6 by dchan@chromium.org, Oct 7 2016

Labels: VerifyIn-55

Comment 7 by dchan@chromium.org, Oct 10 2016

Labels: -VerifyIn-55

Comment 8 by dchan@google.com, Nov 19 2016

Labels: VerifyIn-56

Comment 9 by dchan@google.com, Jan 21 2017

Labels: VerifyIn-57

Comment 10 by dchan@google.com, Mar 4 2017

Labels: VerifyIn-58

Comment 11 by dchan@google.com, Apr 17 2017

Labels: VerifyIn-59

Comment 12 by dchan@google.com, May 30 2017

Labels: VerifyIn-60
Labels: VerifyIn-61

Comment 14 by dchan@chromium.org, Oct 14 2017

Status: Archived (was: Fixed)

Sign in to add a comment