New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 673966 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Apr 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Feature



Sign in to add a comment

rebaseline-cl: Support "filling in" missing platforms.

Project Member Reported by qyears...@chromium.org, Dec 14 2016

Issue description

Currently, rebaseline-cl requires results from every supported platform.

If results from every platform can always be obtained in a reasonable amount of time, then this is ideal, since it means that the new baselines should always be accurate for every platform.

But, if/when results for a particular platform are not available, due to flaky failures or busy try bots, etc., the results for some platforms can usually be guessed based on results from other platforms.

For example, if mac10.11_retina_blink_rel fails for some reason, but all of the other platforms have results, then we could guess that the results are the same as mac10.11_blink_rel (which is the builder for the next port in the "fallback path").
 
Cc: chrishtr@chromium.org
Labels: -Pri-3 Pri-2
Owner: qyears...@chromium.org
Status: Assigned (was: Available)
Chris hit a case where twice the android bot failed, so webkit-patch rebaseline-cl was refusing to rebaseline, but we didn't actually need results from the android bot, since the test isn't run on the android bot.

I think this might be a fairly common case.

Also, I think there are a couple ways that the tool could be changed:
 1. Add an option --ignore-missing-builds which should make it fill in any missing results.
 2. Add an option --ignore-builder=foo, to ignore a specific builder.
My guess is that the first option might be easier to use...
Cc: dpranke@chromium.org zakerinasab@chromium.org
 Issue 658880  has been merged into this issue.
Notes:

My initial approach: After fetching results for try jobs, there's a dict mapping builds to LayoutTestResults objects; I was going to try to just fill in that dict by copying LayoutTestResults from some builds to others.

This doesn't actually work, since the LayoutTestResults objects are just used to get the list of baselines to download; when actually downloading the baselines, the command `webkit-patch rebaseline-test-internal --builder=... --build_number=...` is used, and if there are no actual results for a builder, then this will fail.

Next approaches to try:

 (1) When filling in results, build up the test_prefix_list with only the builders that have results; then, after calling self.rebaseline (https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py?l=102), actually try to copy files over to different locations.

 (2) Modify webkit-patch rebaseline-test-internal to take an optional other port name, which will override the port which would otherwise be used to determine the baseline directory, based on the actual builder name. Then, change the test_prefix_list to contain these overriding port names.

Comment 4 by drott@chromium.org, Mar 24 2017

Cc: drott@chromium.org

Comment 5 by drott@chromium.org, Mar 24 2017

Does (1) from the list above mean that the results are copied to more specific platform locations? Couldn't that be an approach if other ports are missing and, optimize/dedupe later, if someone rebaselines this test case again and additional results are available?



Status: Started (was: Assigned)
(1) from the above list would mean that, before de-duping, baseline files are copied (e.g. from linux/ to android/) and then deduped.

Now, though, I think that (2) would be simpler to implement; that approach requires that first the rebaseline-test-internal command support taking a separate builder and port name (https://codereview.chromium.org/2776543003).
Project Member

Comment 7 by bugdroid1@chromium.org, Mar 24 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3c6862c764ace98c3075f88f862f888bad739494

commit 3c6862c764ace98c3075f88f862f888bad739494
Author: qyearsley <qyearsley@chromium.org>
Date: Fri Mar 24 22:17:11 2017

Change rebaseline-test-internal to take port name option.

This CL changes `webkit-patch rebaseline-test-internal` to take a port
name in addition to a builder name. The builder name determines where
to pull the result from, and the port name determines what port the
new baseline belongs to.

The purpose of this change is to make it so that that baselines can
be downloaded from a builder that does not necessarily correspond
to the port that the baseline is for (see point 2 in  http://crbug.com/673966#c3 ).

In addition, this CL also changes some related things for consistency:
 - webkit-patch copy-existing-baselines-internal takes a port instead
   of builder name; previously it had just used the builder name
   to get a Port.
 - The order of the arguments passed to these functions is changed and
   made more consistent.
 - The formatting of the command arguments in the tests is cleaned up.
 - The MockLineRemovingExecutive in rebaseline_unittest is changed.

BUG= 673966 

Review-Url: https://codereview.chromium.org/2776543003
Cr-Commit-Position: refs/heads/master@{#459573}

[modify] https://crrev.com/3c6862c764ace98c3075f88f862f888bad739494/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/auto_rebaseline_unittest.py
[modify] https://crrev.com/3c6862c764ace98c3075f88f862f888bad739494/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/optimize_baselines.py
[modify] https://crrev.com/3c6862c764ace98c3075f88f862f888bad739494/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py
[modify] https://crrev.com/3c6862c764ace98c3075f88f862f888bad739494/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py
[modify] https://crrev.com/3c6862c764ace98c3075f88f862f888bad739494/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_unittest.py

Project Member

Comment 8 by bugdroid1@chromium.org, Apr 5 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/339734d88bb8df1cc19b7f0bee7cdcf0645fb1ef

commit 339734d88bb8df1cc19b7f0bee7cdcf0645fb1ef
Author: qyearsley <qyearsley@chromium.org>
Date: Wed Apr 05 04:44:39 2017

Add support for rebaselining for a port from a non-corresponding builder.

This CL changes TestBaselineSet to store port names as well as builds,
and makes it so that the rebaseline method, when given constructing a
list of commands to invoke for a given TestBaselineSet, the port name
is according to the TestBaselineSet and doesn't necessarily need to
correspond to the builder name.

After this change, I believe that the only change left before
rebaseline-cl supports rebaselining with only some try bot results
available is to change the way that TestBaselineSets are created
in rebaseline_cl.py.

BUG= 673966 

Review-Url: https://codereview.chromium.org/2790863002
Cr-Commit-Position: refs/heads/master@{#461980}

[modify] https://crrev.com/339734d88bb8df1cc19b7f0bee7cdcf0645fb1ef/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py
[modify] https://crrev.com/339734d88bb8df1cc19b7f0bee7cdcf0645fb1ef/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_unittest.py

Project Member

Comment 9 by bugdroid1@chromium.org, Apr 13 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ed4ab0401c27e0643fa370fc3d61e5781099ab69

commit ed4ab0401c27e0643fa370fc3d61e5781099ab69
Author: qyearsley <qyearsley@chromium.org>
Date: Thu Apr 13 22:52:54 2017

rebaseline-cl: Enable filling in missing results if no results could be fetched.

The previous CL (https://codereview.chromium.org/2803143002) added most of
the logic for filling in missing results, but for cases where a build
exist but has no results (e.g. infra failures, unfinished builds), the
rebaseline-cl command would still abort; this CL changes it so that if
--fill-missing is given, rebaseline-cl will still continue after results
were not found for some existing builds.

BUG= 673966 

Review-Url: https://codereview.chromium.org/2817583004
Cr-Commit-Position: refs/heads/master@{#464587}

[modify] https://crrev.com/ed4ab0401c27e0643fa370fc3d61e5781099ab69/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py
[modify] https://crrev.com/ed4ab0401c27e0643fa370fc3d61e5781099ab69/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py

Project Member

Comment 10 by bugdroid1@chromium.org, Apr 14 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8f030f1d89d3a3ccc4bcf8cc687a7ad6f413256c

commit 8f030f1d89d3a3ccc4bcf8cc687a7ad6f413256c
Author: qyearsley <qyearsley@chromium.org>
Date: Fri Apr 14 04:20:10 2017

rebaseline-cl: Add logic for selecting a related builder when filling in results.

This is a follow-up to https://codereview.chromium.org/2803143002 which adds
some logic for deciding what builder to use when we're trying to rebaseline
from try jobs and platforms have no results available.

Initially, the logic was: just choose any build (the first build).
This would change it to: prefer choosing some build with the same basic OS.

BUG= 673966 

Review-Url: https://codereview.chromium.org/2808043005
Cr-Commit-Position: refs/heads/master@{#464675}

[modify] https://crrev.com/8f030f1d89d3a3ccc4bcf8cc687a7ad6f413256c/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py
[modify] https://crrev.com/8f030f1d89d3a3ccc4bcf8cc687a7ad6f413256c/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py

Status: Fixed (was: Started)
Support should be added now! But it's not yet verified or used in practice. Now after https://codereview.chromium.org/2822613003, this should be the behavior used when doing web-platform-tests imports.
Project Member

Comment 12 by bugdroid1@chromium.org, Sep 20 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5e2158cebad5831d103a3bfdb85e9969b7275198

commit 5e2158cebad5831d103a3bfdb85e9969b7275198
Author: Daniel Cheng <dcheng@chromium.org>
Date: Wed Sep 20 06:46:53 2017

Remove use-chromium-style-naming option from BlinkGCPlugin

Bug:  673966 
Change-Id: If0991048ed26f584f3a7fda081d561dc3f30f194
Reviewed-on: https://chromium-review.googlesource.com/674283
Reviewed-by: Kentaro Hara <haraken@chromium.org>
Commit-Queue: Daniel Cheng <dcheng@chromium.org>
Cr-Commit-Position: refs/heads/master@{#503073}
[modify] https://crrev.com/5e2158cebad5831d103a3bfdb85e9969b7275198/tools/clang/blink_gc_plugin/BlinkGCPlugin.cpp

Sign in to add a comment