[cros bisect] Need to continue bisecting when a test run fails to run. |
||||||||
Issue descriptionWe were attempting to bisect a Chrome regression ( crbug.com/766385 ) using the cros bisect script. The bisect script stopped after running one test. The test's metric is "Failures_tabopenclose" and it's json result file is initialized before the the test runs. If the test raises an exception, we consider that a valid failure and would like to continue on with the bisection. However, the bisect script stops when an exception is raised and does not look at the result file. Repro command: cros bisect -B 3cd860762dd1b4b595c959c2dfff16f84fa99d3d -G 4cd1e85e769baf04e01496fa5047a37f455c47b9 -r ${IP} -b ${BOARD} --repo chromium --chromium-dir /usr/local/google/home/gurchetansingh/browser --evaluator autotest --reuse-repo --test-name graphics_Stress.tabopenclose --metric Failures_tabopenclose -d ${RESULT_DIR} --reuse-build --reuse-repo --reuse-eval 2>&1 | tee output.log
,
Sep 25 2017
Investigating...
,
Sep 25 2017
It should be a feature request that "cros bisect" treats failed run as a bad commit (or even look at the test result if it has one) instead of stopping bisect. This is reasonable and doable. Working on it.
,
Sep 25 2017
https://bugs.chromium.org/p/chromium/issues/detail?id=763834 is also a good test case.
,
Sep 27 2017
CL under review: https://chromium-review.googlesource.com/c/chromiumos/chromite/+/686025/2 Gurchetan, you may test it on your side. I tested it and it can bisect the culprit of your example.
,
Sep 28 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/586820c53abe2913e8a441b81a2ec834ad9c30bb commit 586820c53abe2913e8a441b81a2ec834ad9c30bb Author: Dean Liao <deanliao@chromium.org> Date: Thu Sep 28 18:18:54 2017 [cros bisect] Retry cros flash. Sometiems "cros flash" fails transiently. Adds retry logic on it. BUG=chromium:766822 TEST=unittest Change-Id: I41f4906b509a7ebc152c0e8c7d94bcb1f4893d37 Reviewed-on: https://chromium-review.googlesource.com/686514 Commit-Ready: Shuo-Peng Liao <deanliao@chromium.org> Tested-by: Shuo-Peng Liao <deanliao@chromium.org> Reviewed-by: Shuo-Peng Liao <deanliao@chromium.org> [modify] https://crrev.com/586820c53abe2913e8a441b81a2ec834ad9c30bb/cli/cros/cros_bisect.py [modify] https://crrev.com/586820c53abe2913e8a441b81a2ec834ad9c30bb/cros_bisect/chrome_on_cros_bisector.py [modify] https://crrev.com/586820c53abe2913e8a441b81a2ec834ad9c30bb/cros_bisect/chrome_on_cros_bisector_unittest.py
,
Oct 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/f558f9c9f09759edd34fceaff041157796b78d2c commit f558f9c9f09759edd34fceaff041157796b78d2c Author: Dean Liao <deanliao@chromium.org> Date: Tue Oct 24 05:39:05 2017 [cros bisect] Continue bisect when autotest returns error. Instead of stopping bisect, it now can be set to continue running even when an autotest evaluation returns error. Example: cros bisect -G R63-9943.0.0 -B R63-9944.0.0 -r $DUT_IP -d $BASE_DIR \ --cros-dir $CROS_DIR --reuse --auto-threshold \ --test-name graphics_Stress.tabopenclose \ --metric Failures_tabopenclose/summary/value \ --log-level info \ --eval-repeat 1 --eval-failsafe --eval-raise-on-error BUG=chromium:766822 TEST=unittest and manaual regression test (see above.) Change-Id: I887b5b4dcda9fb7c3edf2bcc8ae9c830d3e18dfe Reviewed-on: https://chromium-review.googlesource.com/686025 Commit-Ready: Shuo-Peng Liao <deanliao@chromium.org> Tested-by: Shuo-Peng Liao <deanliao@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [modify] https://crrev.com/f558f9c9f09759edd34fceaff041157796b78d2c/cros_bisect/git_bisector.py [modify] https://crrev.com/f558f9c9f09759edd34fceaff041157796b78d2c/cros_bisect/autotest_evaluator_unittest.py [modify] https://crrev.com/f558f9c9f09759edd34fceaff041157796b78d2c/cros_bisect/git_bisector_unittest.py [modify] https://crrev.com/f558f9c9f09759edd34fceaff041157796b78d2c/cros_bisect/chrome_on_cros_bisector_unittest.py [modify] https://crrev.com/f558f9c9f09759edd34fceaff041157796b78d2c/cros_bisect/autotest_evaluator.py [modify] https://crrev.com/f558f9c9f09759edd34fceaff041157796b78d2c/cli/cros/cros_bisect.py
,
Nov 13 2017
Found another case that because of a incompatible Chrome (it can be built and deployed to DUT successfully. But when running autotest, the Chrome crashed), cros bisect terminated because of unhandled exception (SSH exception). cros bisect -G R64-10070.0.0 -B R64-10071.0.0 -r 192.168.0.16 -d /disks/ssd/bisect_773578 --auto-threshold --test-name graphics_WebGLAium --metric avg_fps_1000_fishes/summary/value --log-level info --eval-repeat 1 --reuse-repo --reuse-build --skip-failed-commit
,
Nov 13 2017
--test-name should be graphics_WebGLAquarium cros bisect -G R64-10070.0.0 -B R64-10071.0.0 -r 192.168.0.16 -d /disks/ssd/bisect_773578 --auto-threshold --test-name graphics_WebGLAquarium --metric avg_fps_1000_fishes/summary/value --log-level info --eval-repeat 1 --reuse-repo --reuse-build --skip-failed-commit --debug
,
Nov 13 2017
,
Feb 14 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/1ae2b146e7dda82f868f4f124c8a2a70d94c1f74 commit 1ae2b146e7dda82f868f4f124c8a2a70d94c1f74 Author: Dean Liao <deanliao@chromium.org> Date: Wed Feb 14 12:07:00 2018 [bisect] Deal with build/deploy/eval failure. Improve cros bisect's robustness by handling Chrome build and deploy failure as well as autotest evaluation failure. When failure occurs it can either raise an exception or just mark the commit as skip. BUG=chromium:766822 TEST=None Change-Id: I9013539fad2df1834f7bd3544ab06ccdad1829bc Reviewed-on: https://chromium-review.googlesource.com/765641 Commit-Ready: Shuo-Peng Liao <deanliao@chromium.org> Tested-by: Shuo-Peng Liao <deanliao@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [modify] https://crrev.com/1ae2b146e7dda82f868f4f124c8a2a70d94c1f74/cros_bisect/git_bisector.py [modify] https://crrev.com/1ae2b146e7dda82f868f4f124c8a2a70d94c1f74/cros_bisect/simple_chrome_builder.py [modify] https://crrev.com/1ae2b146e7dda82f868f4f124c8a2a70d94c1f74/cros_bisect/autotest_evaluator_unittest.py [modify] https://crrev.com/1ae2b146e7dda82f868f4f124c8a2a70d94c1f74/cros_bisect/simple_chrome_builder_unittest.py [modify] https://crrev.com/1ae2b146e7dda82f868f4f124c8a2a70d94c1f74/cros_bisect/chrome_on_cros_bisector.py [modify] https://crrev.com/1ae2b146e7dda82f868f4f124c8a2a70d94c1f74/cros_bisect/builder.py [modify] https://crrev.com/1ae2b146e7dda82f868f4f124c8a2a70d94c1f74/cros_bisect/git_bisector_unittest.py [modify] https://crrev.com/1ae2b146e7dda82f868f4f124c8a2a70d94c1f74/cros_bisect/chrome_on_cros_bisector_unittest.py [modify] https://crrev.com/1ae2b146e7dda82f868f4f124c8a2a70d94c1f74/cros_bisect/autotest_evaluator.py [modify] https://crrev.com/1ae2b146e7dda82f868f4f124c8a2a70d94c1f74/cli/cros/cros_bisect.py
,
Feb 14 2018
Removing Speed>Bisection component since this is unrelated to Pinpoint or recipe bisect.
,
May 3 2018
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by pwang@chromium.org
, Sep 19 2017