TastVMTest failures are reported poorly |
||
Issue descriptionThe results of Tast VM test failures in the betty-arc64-paladin build at https://ci.chromium.org/p/chromeos/builders/luci.chromeos.general/CQ/b8930293407931948512 are poorly displayed. In the "TastVMTest (attempt 1)" stage, the network.ShillStability test failed (and is listed in the stage), but the stage itself is green. It seems like it should be red, or maybe yellow (since the stage is retried on failure). stdout for this stage is also confusing: ... 2018/11/09 18:44:37 network.ShillStability [ FAIL ] Found 2 shill processes ([5716 5776]); want 1 ... 2018/11/09 18:44:37 -------------------------------------------------------------------------------- 2018/11/09 18:44:37 Results saved to /tmp/cbuildbotBK2ntD/tast_vm_paladin 18:44:37: INFO: Tests succeeded. 18:44:37: DEBUG: Stop VM 18:44:37: INFO: Killing 266. 18:44:37: INFO: RunCommand: kill -9 266 18:44:38: DEBUG: Stop VM 18:44:38: DEBUG: /tmp/cros_vm_9222 not present. 18:44:38: INFO: Time elapsed 0:04:30.207295. 18:44:39: INFO: Uploading artifacts to Cloud Storage... ... [1;31m18:44:41: ERROR: 1 test(s) failed[0m [1;31m18:44:41: ERROR: Tast VM tests failed Traceback (most recent call last): File "/b/swarming/wTuPS3R/ir/cache/cbuild/repository/chromite/cbuildbot/stages/tast_test_stages.py", line 119, in PerformStage got_exception) File "/b/swarming/wTuPS3R/ir/cache/cbuild/repository/chromite/lib/failures_lib.py", line 230, in wrapped_functor return functor(*args, **kwargs) File "/b/swarming/wTuPS3R/ir/cache/cbuild/repository/chromite/cbuildbot/stages/tast_test_stages.py", line 229, in _ProcessAndArchiveResults raise e TestFailure: ** 1 test(s) failed **[0m [1;33m18:44:41: WARNING: ** 1 test(s) failed **[0m --- I'd like to fix the "Tests succeeded" message, as it seems bound to confuse people. It's coming from chromite/scripts/cros_run_vm_test.py, which assumes that a zero status code means that all tests passed: def _OutputResults(self, result): ... result_str = 'succeeded' if result.returncode == 0 else 'failed' logging.info('Tests %s.', result_str) (Aside: please don't interpolate trivial strings like this, as it makes it extremely hard to find the code that's responsible for logging the message.) (I should also figure out why network.ShillStability failed.)
,
Nov 16
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/8f54c42bc4028ac0d619ae5f67a7bae33ecf68de commit 8f54c42bc4028ac0d619ae5f67a7bae33ecf68de Author: Daniel Erat <derat@chromium.org> Date: Fri Nov 16 13:02:20 2018 scripts: Improve status logging in cros_run_vm_test.py. Update cros_run_vm_test.py to log "___ exited with status code ___." It previously always logged either "Tests succeeded." or "Tests failed.", which could be confusing when running other commands: for example, the "tast run" command executed by the TastVMTest stage exits with 0 even if individual Tast tests failed (since the stage is responsible for interpreting the results). BUG= chromium:904110 TEST=ran tryjob and checked logs Change-Id: I36555202222533d8a3f9a7544d879057665b0477 Reviewed-on: https://chromium-review.googlesource.com/1330777 Commit-Ready: Dan Erat <derat@chromium.org> Tested-by: Dan Erat <derat@chromium.org> Reviewed-by: Achuith Bhandarkar <achuith@chromium.org> Reviewed-by: Ben Pastene <bpastene@chromium.org> [modify] https://crrev.com/8f54c42bc4028ac0d619ae5f67a7bae33ecf68de/scripts/cros_run_vm_test.py
,
Nov 20
Logging should be improved now. I haven't noticed the green-for-retry stage issue again -- all of the recent TastVMTest failures I can find are yellow, with the retry appearing as red (due to issue 900804 ). |
||
►
Sign in to add a comment |
||
Comment 1 by derat@chromium.org
, Nov 10