rebaseline-cl should download baselines for newly-added tests. |
||||
Issue descriptionCurrently, by default, the webkit-patch rebaseline-cl command gets the list of tests to rebaseline by seeing what tests had mismatches with existing baselines in some try job results. It should also download baselines for new tests. For example, the CL http://crrev.com/2401363003 added editing/input/caret-read-only-after-editable.html; running webkit-patch rebaseline-cl with no extra arguments didn't download the new baseline. In the test results it's being spotted as a new test: https://storage.googleapis.com/chromium-layout-test-archives/linux_precise_blink_rel/3620/layout-test-results/results.html We should be able to tell if it's a new test if the "actual" test result in the JSON is "MISSING", and in general, we should download baselines for tests with "MISSING" results.
,
Oct 11 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/aa5cab9d5435feb0098f217286a43f02c63ff182 commit aa5cab9d5435feb0098f217286a43f02c63ff182 Author: qyearsley <qyearsley@chromium.org> Date: Tue Oct 11 22:19:11 2016 In rebaseline-cl, also rebaseline tests with MISSING results. Reason: Newly-added tests have the test result MISSING in try jobs; we also want to download new baselines when possible for tests with the actual result MISSING, not just tests with results that don't match existing baselines. In theory this should make it so that when webkit-patch rebaseline-cl is run with no extra args, baselines for newly-added tests should also be downloaded. BUG= 654604 Review-Url: https://codereview.chromium.org/2409343003 Cr-Commit-Position: refs/heads/master@{#424568} [modify] https://crrev.com/aa5cab9d5435feb0098f217286a43f02c63ff182/third_party/WebKit/Tools/Scripts/webkitpy/common/net/layouttestresults.py [modify] https://crrev.com/aa5cab9d5435feb0098f217286a43f02c63ff182/third_party/WebKit/Tools/Scripts/webkitpy/common/net/layouttestresults_unittest.py [modify] https://crrev.com/aa5cab9d5435feb0098f217286a43f02c63ff182/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py [modify] https://crrev.com/aa5cab9d5435feb0098f217286a43f02c63ff182/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py
,
Oct 11 2016
,
Oct 12 2016
Result of the above test: 1. When I ran webkit-patch rebaseline-cl, new baselines for the new test were downloaded, in addition to new baselines for some flaky tests that I don't care about. 2. I reran with webkit-patch rebaseline-cl --only-changed-tests, which behaved as expected.
,
Oct 26 2016
Just found a mistake in the above change: rebaseline-cl should only rebaseline tests with missing results if this is unexpected; if there are test expectation lines with [ Missing ] as the expected result, then these shouldn't be rebaselined.
,
Nov 1 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/7526dc346a39764341c162734d57e48cea499ea2 commit 7526dc346a39764341c162734d57e48cea499ea2 Author: qyearsley <qyearsley@chromium.org> Date: Tue Nov 01 22:35:51 2016 rebaseline-cl: Don't rebaseline expected-MISSING tests. Before this CL: If there are tests with [ Missing ] expectations in TestExpectations, then `webkit-patch rebaseline-cl` will download new baselines for those tests. I think this behavior isn't quite what I think we want, since any such tests are probably not directly related to the CL for which new baselines are being downloaded. After this CL: `webkit-patch rebaseline-cl` will only download baselines for tests that have unexpected results (either unexpectedly mismatching baselines or unexpectedly missing baselines). BUG= 654604 Review-Url: https://codereview.chromium.org/2453223002 Cr-Commit-Position: refs/heads/master@{#429138} [modify] https://crrev.com/7526dc346a39764341c162734d57e48cea499ea2/third_party/WebKit/Tools/Scripts/webkitpy/common/net/layouttestresults.py [modify] https://crrev.com/7526dc346a39764341c162734d57e48cea499ea2/third_party/WebKit/Tools/Scripts/webkitpy/common/net/layouttestresults_unittest.py [modify] https://crrev.com/7526dc346a39764341c162734d57e48cea499ea2/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py
,
Nov 2 2016
|
||||
►
Sign in to add a comment |
||||
Comment 1 by qyears...@chromium.org
, Oct 10 2016Labels: -OS-Linux