Issue metadata
Sign in to add a comment
|
run-webkit-tests repeat/iterations should be listed in actual results similar to retries |
||||||||||||||||||||||
Issue descriptionIn the JSON output of r-w-t, each test has an "actual" field which may contain multiple results if the test is retried, e.g. "FAIL TIMEOUT PASS". However, when --repeat-each or --iterations is specified, the "actual" field only contains the last result. It'd more intuitive to have all the results listed, similar to the retry behaviour. It's also crucial for FindIt's flakiness detector. @Quinten, do you know if this was intentional? Would I break anything if I change this behaviour? (AFAIK, no bots currently use --repeat or --iterations.)
,
Apr 3 2018
,
Apr 13 2018
Sorry, there were some infra firefightings I needed to do earlier this week. And I just started working on it today. I'm now hoping to get it done next week, but I'll be at BlinkOn so can't really promise.
,
Apr 20 2018
The NextAction date has arrived: 2018-04-20
,
Apr 26 2018
,
Apr 26 2018
,
Apr 30 2018
Please cc' seanmccullough@ and myself on any changes that might affect the json test results format.
,
Apr 30 2018
,
May 3 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/6d938afc56b4b612ae3ea83e4734f3a72418b4a8 commit 6d938afc56b4b612ae3ea83e4734f3a72418b4a8 Author: Robert Ma <robertma@chromium.org> Date: Thu May 03 19:48:16 2018 [blinkpy] List all repeats & iterations in actual results Previously, if a test was retried (because of unexpected failures), the results from all retry attempts would show up in the "actual" field in the summary; but if a test ran multiple times because of --iterations or --repeat-each, only the last result would be included in the summary. This CL makes iterations and repeats behave the same as retries in terms of the result summary. All results from all iterations and repeats from all retries are now included in the "actual" result field in summary. This will make it easier for other tools (e.g. FindIt) and users to debug flaky tests, and won't affect existing bots as they don't use --iterations or --repeat-each. Some code-health improvements are done along the way (e.g. expectation_to_string). There are some questions regarding the existing behaviours. In order to minimize the risk, this CL strives to ensure backward compatibility. We will revisit some of the behaviours later. Bug: 828605 Change-Id: Ibb5a329eb8dc60fd0ae857083ebddc3b1c86f5f0 Reviewed-on: https://chromium-review.googlesource.com/1029430 Commit-Queue: Robert Ma <robertma@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Quinten Yearsley <qyearsley@chromium.org> Cr-Commit-Position: refs/heads/master@{#555848} [modify] https://crrev.com/6d938afc56b4b612ae3ea83e4734f3a72418b4a8/third_party/blink/tools/blinkpy/web_tests/controllers/manager.py [modify] https://crrev.com/6d938afc56b4b612ae3ea83e4734f3a72418b4a8/third_party/blink/tools/blinkpy/web_tests/layout_package/bot_test_expectations.py [modify] https://crrev.com/6d938afc56b4b612ae3ea83e4734f3a72418b4a8/third_party/blink/tools/blinkpy/web_tests/models/test_expectations.py [modify] https://crrev.com/6d938afc56b4b612ae3ea83e4734f3a72418b4a8/third_party/blink/tools/blinkpy/web_tests/models/test_run_results.py [modify] https://crrev.com/6d938afc56b4b612ae3ea83e4734f3a72418b4a8/third_party/blink/tools/blinkpy/web_tests/models/test_run_results_unittest.py [modify] https://crrev.com/6d938afc56b4b612ae3ea83e4734f3a72418b4a8/third_party/blink/tools/blinkpy/web_tests/run_webkit_tests.py [modify] https://crrev.com/6d938afc56b4b612ae3ea83e4734f3a72418b4a8/third_party/blink/tools/blinkpy/web_tests/run_webkit_tests_unittest.py
,
May 3 2018
@chanli could you trigger some jobs with repeat to verify? Thanks!
,
May 31 2018
|
|||||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||||
Comment 1 by robertma@chromium.org
, Apr 3 2018