New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 654604 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Feature

Blocking:
issue 474273



Sign in to add a comment

rebaseline-cl should download baselines for newly-added tests.

Project Member Reported by qyears...@chromium.org, Oct 10 2016

Issue description

Currently, by default, the webkit-patch rebaseline-cl command gets the list of tests to rebaseline by seeing what tests had mismatches with existing baselines in some try job results. It should also download baselines for new tests.

For example, the CL http://crrev.com/2401363003 added editing/input/caret-read-only-after-editable.html; running webkit-patch rebaseline-cl with no extra arguments didn't download the new baseline.

In the test results it's being spotted as a new test:

https://storage.googleapis.com/chromium-layout-test-archives/linux_precise_blink_rel/3620/layout-test-results/results.html

We should be able to tell if it's a new test if the "actual" test result in the JSON is "MISSING", and in general, we should download baselines for tests with "MISSING" results.
 
Cc: wkorman@chromium.org
Labels: -OS-Linux
Project Member

Comment 2 by bugdroid1@chromium.org, Oct 11 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/aa5cab9d5435feb0098f217286a43f02c63ff182

commit aa5cab9d5435feb0098f217286a43f02c63ff182
Author: qyearsley <qyearsley@chromium.org>
Date: Tue Oct 11 22:19:11 2016

In rebaseline-cl, also rebaseline tests with MISSING results.

Reason: Newly-added tests have the test result MISSING in try jobs; we also want to download new baselines when possible for tests with the actual result MISSING, not just tests with results that don't match existing baselines.

In theory this should make it so that when webkit-patch rebaseline-cl is run with no extra args, baselines for newly-added tests should also be downloaded.

BUG= 654604 

Review-Url: https://codereview.chromium.org/2409343003
Cr-Commit-Position: refs/heads/master@{#424568}

[modify] https://crrev.com/aa5cab9d5435feb0098f217286a43f02c63ff182/third_party/WebKit/Tools/Scripts/webkitpy/common/net/layouttestresults.py
[modify] https://crrev.com/aa5cab9d5435feb0098f217286a43f02c63ff182/third_party/WebKit/Tools/Scripts/webkitpy/common/net/layouttestresults_unittest.py
[modify] https://crrev.com/aa5cab9d5435feb0098f217286a43f02c63ff182/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py
[modify] https://crrev.com/aa5cab9d5435feb0098f217286a43f02c63ff182/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py

Status: Started (was: Assigned)
Now testing this out: http://crrev.com/2405213004.
Status: Fixed (was: Started)
Result of the above test:

1. When I ran webkit-patch rebaseline-cl, new baselines for the new test were downloaded, in addition to new baselines for some flaky tests that I don't care about.
2. I reran with webkit-patch rebaseline-cl --only-changed-tests, which behaved as expected.
Status: Started (was: Fixed)
Just found a mistake in the above change: rebaseline-cl should only rebaseline tests with missing results if this is unexpected; if there are test expectation lines with [ Missing ] as the expected result, then these shouldn't be rebaselined.
Project Member

Comment 6 by bugdroid1@chromium.org, Nov 1 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7526dc346a39764341c162734d57e48cea499ea2

commit 7526dc346a39764341c162734d57e48cea499ea2
Author: qyearsley <qyearsley@chromium.org>
Date: Tue Nov 01 22:35:51 2016

rebaseline-cl: Don't rebaseline expected-MISSING tests.

Before this CL: If there are tests with [ Missing ] expectations in TestExpectations, then `webkit-patch rebaseline-cl` will download new baselines for those tests. I think this behavior isn't quite what I think we want, since any such tests are probably not directly related to the CL for which new baselines are being downloaded.

After this CL: `webkit-patch rebaseline-cl` will only download baselines for tests that have unexpected results (either unexpectedly mismatching baselines or unexpectedly missing baselines).

BUG= 654604 

Review-Url: https://codereview.chromium.org/2453223002
Cr-Commit-Position: refs/heads/master@{#429138}

[modify] https://crrev.com/7526dc346a39764341c162734d57e48cea499ea2/third_party/WebKit/Tools/Scripts/webkitpy/common/net/layouttestresults.py
[modify] https://crrev.com/7526dc346a39764341c162734d57e48cea499ea2/third_party/WebKit/Tools/Scripts/webkitpy/common/net/layouttestresults_unittest.py
[modify] https://crrev.com/7526dc346a39764341c162734d57e48cea499ea2/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py

Status: Fixed (was: Started)

Sign in to add a comment