Build failed but reported as Pass |
||||||||
Issue descriptionVersion: M52 false build status report as pass OS: chromeOS What steps will reproduce the problem? (1) goto https://cros-goldeneye.corp.google.com/console/viewBuild?version=8302.0.0&channel=dev&type=release (2) click on Mccloud 'Pass' under 'Build status' (3) will take you to https://uberchromegw.corp.google.com/i/chromeos/builders/mccloud-release/builds/12 What is the expected output? build page in #3 should be all green, since #2 said so. What do you see instead? build failure Please use labels and text to provide additional information. I have not check all of them but here are a few more Panther https://uberchromegw.corp.google.com/i/chromeos/builders/panther-release/builds/631 Link https://uberchromegw.corp.google.com/i/chromeos/builders/link-release/builds/4439 From Christine Lee It does look like the build is marked a failure in the waterfall, but it isn't marked a failure in the build metadata here: https://storage.cloud.google.com/chromeos-image-archive/mccloud-release/R52-8297.0.0/metadata.json.
,
May 11 2016
,
May 11 2016
,
May 13 2016
,
May 18 2016
The status of 'Paygen' is still 'inflight' in buildStageTable, and the result of 'Paygen' stage wasn't added to results_lib.Results. Probably the Paygen thread got killed in the middle and the finally block of the PaygenStage didn't get executed? The reportStage only checks results_lib.Results.BuildSucceededSoFar(), which doesn't contain the Paygen status in this case, so it thinks the final_status is 'pass' and marks this build as 'pass' in buildTable. Didn't find the other informational logs about the PaygenStage, shall we turn some logs to info/warning in paralell.py when it's trying to kill tasks?
,
May 18 2016
,
May 21 2016
,
May 27 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/fdf3781fc64e86dfe20a702472a44ad72177b41b commit fdf3781fc64e86dfe20a702472a44ad72177b41b Author: Ningning Xia <nxia@chromium.org> Date: Thu May 19 18:37:06 2016 Verify stage status from CIDB for results_lib.Result. 1.When timeout happens and background tasks are killed, the statuses of the background stage tasks may get lost. BuildSucceededSoFar may still return build_succeess = True when the killed stage tasks were failed. So add one more verification step in _BuildSucceededFromCIDB to check the stage status in CIDB. 2.Added error logs to see why the failed status cannot be recorded correctly. 3.Record failure Result when WaitUntilReady returns False. BUG= chromium:610776 TEST=unit test/trybot Change-Id: Iae14282559203429a73b00da83c8f0c230a3be61 Reviewed-on: https://chromium-review.googlesource.com/346201 Commit-Ready: Ningning Xia <nxia@chromium.org> Tested-by: Ningning Xia <nxia@chromium.org> Reviewed-by: Ningning Xia <nxia@chromium.org> [modify] https://crrev.com/fdf3781fc64e86dfe20a702472a44ad72177b41b/cbuildbot/constants.py [modify] https://crrev.com/fdf3781fc64e86dfe20a702472a44ad72177b41b/cbuildbot/stages/generic_stages.py [modify] https://crrev.com/fdf3781fc64e86dfe20a702472a44ad72177b41b/lib/fake_cidb.py [modify] https://crrev.com/fdf3781fc64e86dfe20a702472a44ad72177b41b/cbuildbot/builders/generic_builders.py [modify] https://crrev.com/fdf3781fc64e86dfe20a702472a44ad72177b41b/cbuildbot/results_lib_unittest.py [modify] https://crrev.com/fdf3781fc64e86dfe20a702472a44ad72177b41b/cbuildbot/builders/simple_builders.py [modify] https://crrev.com/fdf3781fc64e86dfe20a702472a44ad72177b41b/cbuildbot/results_lib.py [modify] https://crrev.com/fdf3781fc64e86dfe20a702472a44ad72177b41b/cbuildbot/stages/report_stages.py
,
May 31 2016
,
Jun 27 2016
Closing... please feel free to reopen if its not fixed.
,
Jun 27 2016
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by akes...@chromium.org
, May 10 2016