webkit-patch rebaseline-cl does not download results from green tryjobs, incorrectly falling back to other bots |
||||
Issue descriptionI just rebaselined a test using webkit-patch rebaseline-cl, and the result was missing on win10_blink_rel and android_blink_rel. I think the result was incorrect. The whole console output was as below: 2017-09-27 08:52:11,596 - Finished try jobs found for all try bots. 2017-09-27 08:52:11,756 - Failed to fetch results for "win10_blink_rel". 2017-09-27 08:52:11,757 - Results URL: https://storage.googleapis.com/chromium-layout-test-archives/win10_blink_rel/4314/layout-test-results/results.html 2017-09-27 08:52:13,840 - Failed to fetch results for "android_blink_rel". 2017-09-27 08:52:13,841 - Results URL: https://storage.googleapis.com/chromium-layout-test-archives/android_blink_rel/3909/layout-test-results/results.html 2017-09-27 08:52:15,183 - There are some builders with no results: 2017-09-27 08:52:15,183 - android_blink_rel 2017-09-27 08:52:15,183 - win10_blink_rel Would you like to continue? [y/N]: y Would you like to try to fill in missing results with available results? Note: This will generally yield correct results as long as the results are not platform-specific. [y/N]: y 2017-09-27 08:52:31,763 - No retry summary available for "mac10.11_retina_blink_rel". 2017-09-27 08:52:31,924 - No retry summary available for "mac10.10_blink_rel". 2017-09-27 08:52:32,093 - No retry summary available for "mac10.11_blink_rel". 2017-09-27 08:52:32,575 - No retry summary available for "mac10.12_blink_rel". 2017-09-27 08:52:32,583 - For paint/invalidation/outline-change-in-scrollers.html: 2017-09-27 08:52:32,584 - Using "mac10.10_blink_rel" build 4067 for linux-trusty. 2017-09-27 08:52:32,584 - Using "mac10.10_blink_rel" build 4067 for win-win7. 2017-09-27 08:52:32,584 - Using "mac10.10_blink_rel" build 4067 for android-kitkat. 2017-09-27 08:52:32,584 - Using "mac10.10_blink_rel" build 4067 for win-win10. 2017-09-27 08:52:33,094 - Rebaselining paint/invalidation/outline-change-in-scrollers.html I think it's incorrect that the result on mac10.10_blink_rel is used for linux-trusty, win-win7, android-kitkat and win-win10: 1. linux-trusty and win-win7 have results, so we should use their results instead of using mac10.10_blink_rel. 2. android-kitkat and win-win10 should use the results on their fallback chains instead of mac10.10_blink_rel. Actual result: The common baseline was overwritten by the mac result which is different from the common baseline. Expected result: The common baseline should be kept as-is. New baseline should be created for mac.
,
Jan 11 2018
Trying to clarify the problem. So the problem here is if a tryjob is green (all layout tests pass), then rebaseline-cl does not download result from that job. Then the results from these green jobs are considered "missing", incorrectly invocating the fill-missing logic despite that these platforms do have results.
,
Jan 11 2018
As far as I remember, that was the case, but now I think this might no longer be an issue. Now in rebaseline_cl.py in the method _fetch_results there's a comment indicating that we're trying to handle the case where tryjobs are passing: https://chromium.googlesource.com/chromium/src/+/dbe83749/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py#231 I'll try to reproduce this, and close it if the behavior is now correct.
,
Jan 11 2018
Besides the green bot issue, please also verify if the fill-missing logic is correct. Sometimes I see message saying an unrelated fallback bot is used to fill a missing (e.g. mac10.10_blink_rel is used for android). The issue might be just a wrong message instead of wrong behavior.
,
Jan 11 2018
Re #4, so I think fill-missing really is a best-effort thing. The current behaviour is probably not wrong, but can be smarter. See issue 801282 which I just filed.
,
Jan 14 2018
I verified the green bot issue, and as Robert says, the behavior of possibly using a result from one platform to fill in for unrelated platforms is intentional -- sometimes (especially for text baselines) the baseline is supposed to be the same on all platforms. There's a method which guesses which platform might be the best (https://cs.chromium.org/chromium/src/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_cl.py?l=392), but indeed it could be smarter. |
||||
►
Sign in to add a comment |
||||
Comment 1 by qyears...@chromium.org
, Sep 28 2017