rebaseline-cl: Support "filling in" missing platforms. |
||||
Issue descriptionCurrently, rebaseline-cl requires results from every supported platform. If results from every platform can always be obtained in a reasonable amount of time, then this is ideal, since it means that the new baselines should always be accurate for every platform. But, if/when results for a particular platform are not available, due to flaky failures or busy try bots, etc., the results for some platforms can usually be guessed based on results from other platforms. For example, if mac10.11_retina_blink_rel fails for some reason, but all of the other platforms have results, then we could guess that the results are the same as mac10.11_blink_rel (which is the builder for the next port in the "fallback path").
,
Feb 15 2017
,
Mar 17 2017
Notes: My initial approach: After fetching results for try jobs, there's a dict mapping builds to LayoutTestResults objects; I was going to try to just fill in that dict by copying LayoutTestResults from some builds to others. This doesn't actually work, since the LayoutTestResults objects are just used to get the list of baselines to download; when actually downloading the baselines, the command `webkit-patch rebaseline-test-internal --builder=... --build_number=...` is used, and if there are no actual results for a builder, then this will fail. Next approaches to try: (1) When filling in results, build up the test_prefix_list with only the builders that have results; then, after calling self.rebaseline (https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py?l=102), actually try to copy files over to different locations. (2) Modify webkit-patch rebaseline-test-internal to take an optional other port name, which will override the port which would otherwise be used to determine the baseline directory, based on the actual builder name. Then, change the test_prefix_list to contain these overriding port names.
,
Mar 24 2017
,
Mar 24 2017
Does (1) from the list above mean that the results are copied to more specific platform locations? Couldn't that be an approach if other ports are missing and, optimize/dedupe later, if someone rebaselines this test case again and additional results are available?
,
Mar 24 2017
(1) from the above list would mean that, before de-duping, baseline files are copied (e.g. from linux/ to android/) and then deduped. Now, though, I think that (2) would be simpler to implement; that approach requires that first the rebaseline-test-internal command support taking a separate builder and port name (https://codereview.chromium.org/2776543003).
,
Mar 24 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3c6862c764ace98c3075f88f862f888bad739494 commit 3c6862c764ace98c3075f88f862f888bad739494 Author: qyearsley <qyearsley@chromium.org> Date: Fri Mar 24 22:17:11 2017 Change rebaseline-test-internal to take port name option. This CL changes `webkit-patch rebaseline-test-internal` to take a port name in addition to a builder name. The builder name determines where to pull the result from, and the port name determines what port the new baseline belongs to. The purpose of this change is to make it so that that baselines can be downloaded from a builder that does not necessarily correspond to the port that the baseline is for (see point 2 in http://crbug.com/673966#c3 ). In addition, this CL also changes some related things for consistency: - webkit-patch copy-existing-baselines-internal takes a port instead of builder name; previously it had just used the builder name to get a Port. - The order of the arguments passed to these functions is changed and made more consistent. - The formatting of the command arguments in the tests is cleaned up. - The MockLineRemovingExecutive in rebaseline_unittest is changed. BUG= 673966 Review-Url: https://codereview.chromium.org/2776543003 Cr-Commit-Position: refs/heads/master@{#459573} [modify] https://crrev.com/3c6862c764ace98c3075f88f862f888bad739494/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/auto_rebaseline_unittest.py [modify] https://crrev.com/3c6862c764ace98c3075f88f862f888bad739494/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/optimize_baselines.py [modify] https://crrev.com/3c6862c764ace98c3075f88f862f888bad739494/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py [modify] https://crrev.com/3c6862c764ace98c3075f88f862f888bad739494/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py [modify] https://crrev.com/3c6862c764ace98c3075f88f862f888bad739494/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_unittest.py
,
Apr 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/339734d88bb8df1cc19b7f0bee7cdcf0645fb1ef commit 339734d88bb8df1cc19b7f0bee7cdcf0645fb1ef Author: qyearsley <qyearsley@chromium.org> Date: Wed Apr 05 04:44:39 2017 Add support for rebaselining for a port from a non-corresponding builder. This CL changes TestBaselineSet to store port names as well as builds, and makes it so that the rebaseline method, when given constructing a list of commands to invoke for a given TestBaselineSet, the port name is according to the TestBaselineSet and doesn't necessarily need to correspond to the builder name. After this change, I believe that the only change left before rebaseline-cl supports rebaselining with only some try bot results available is to change the way that TestBaselineSets are created in rebaseline_cl.py. BUG= 673966 Review-Url: https://codereview.chromium.org/2790863002 Cr-Commit-Position: refs/heads/master@{#461980} [modify] https://crrev.com/339734d88bb8df1cc19b7f0bee7cdcf0645fb1ef/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py [modify] https://crrev.com/339734d88bb8df1cc19b7f0bee7cdcf0645fb1ef/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_unittest.py
,
Apr 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ed4ab0401c27e0643fa370fc3d61e5781099ab69 commit ed4ab0401c27e0643fa370fc3d61e5781099ab69 Author: qyearsley <qyearsley@chromium.org> Date: Thu Apr 13 22:52:54 2017 rebaseline-cl: Enable filling in missing results if no results could be fetched. The previous CL (https://codereview.chromium.org/2803143002) added most of the logic for filling in missing results, but for cases where a build exist but has no results (e.g. infra failures, unfinished builds), the rebaseline-cl command would still abort; this CL changes it so that if --fill-missing is given, rebaseline-cl will still continue after results were not found for some existing builds. BUG= 673966 Review-Url: https://codereview.chromium.org/2817583004 Cr-Commit-Position: refs/heads/master@{#464587} [modify] https://crrev.com/ed4ab0401c27e0643fa370fc3d61e5781099ab69/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py [modify] https://crrev.com/ed4ab0401c27e0643fa370fc3d61e5781099ab69/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py
,
Apr 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8f030f1d89d3a3ccc4bcf8cc687a7ad6f413256c commit 8f030f1d89d3a3ccc4bcf8cc687a7ad6f413256c Author: qyearsley <qyearsley@chromium.org> Date: Fri Apr 14 04:20:10 2017 rebaseline-cl: Add logic for selecting a related builder when filling in results. This is a follow-up to https://codereview.chromium.org/2803143002 which adds some logic for deciding what builder to use when we're trying to rebaseline from try jobs and platforms have no results available. Initially, the logic was: just choose any build (the first build). This would change it to: prefer choosing some build with the same basic OS. BUG= 673966 Review-Url: https://codereview.chromium.org/2808043005 Cr-Commit-Position: refs/heads/master@{#464675} [modify] https://crrev.com/8f030f1d89d3a3ccc4bcf8cc687a7ad6f413256c/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py [modify] https://crrev.com/8f030f1d89d3a3ccc4bcf8cc687a7ad6f413256c/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl_unittest.py
,
Apr 14 2017
Support should be added now! But it's not yet verified or used in practice. Now after https://codereview.chromium.org/2822613003, this should be the behavior used when doing web-platform-tests imports.
,
Sep 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/5e2158cebad5831d103a3bfdb85e9969b7275198 commit 5e2158cebad5831d103a3bfdb85e9969b7275198 Author: Daniel Cheng <dcheng@chromium.org> Date: Wed Sep 20 06:46:53 2017 Remove use-chromium-style-naming option from BlinkGCPlugin Bug: 673966 Change-Id: If0991048ed26f584f3a7fda081d561dc3f30f194 Reviewed-on: https://chromium-review.googlesource.com/674283 Reviewed-by: Kentaro Hara <haraken@chromium.org> Commit-Queue: Daniel Cheng <dcheng@chromium.org> Cr-Commit-Position: refs/heads/master@{#503073} [modify] https://crrev.com/5e2158cebad5831d103a3bfdb85e9969b7275198/tools/clang/blink_gc_plugin/BlinkGCPlugin.cpp |
||||
►
Sign in to add a comment |
||||
Comment 1 by qyears...@chromium.org
, Jan 25 2017Labels: -Pri-3 Pri-2
Owner: qyears...@chromium.org
Status: Assigned (was: Available)