Wrong results of NeedsRebaselines to produce missing pixel results |
||||
Issue descriptionThe rebaseline https://codereview.chromium.org/2002443003 produce many wrong results. For example: https://codereview.chromium.org/download/issue2002443003_1_10339.diff Actually the text expectations should not be changed at all. The rebaseline was supposed just to produce missing pixel expectations after the tests are converted to pixel tests.
,
May 23 2016
To reproduce #1: 1. gclient sync -r src@8b0b291901ad0e209a616da34f08fb1d045b134d 2. sed -i '/611462/s/Missing/Rebaseline/' third_party/WebKit/LayoutTests/TestExpectations 3. webkit-patch rebaseline-expectations The rebaseline should generate the missing pngs only, but it also generates/modifies some text expectations that are incorrect.
,
May 23 2016
,
Jun 14 2016
Will fix this together with bug 574272 . I think I found the reason: The "latest layout test result" repository on storage server is not a full layout result of the last run, but contains the latest failure of each test of all recent runs. When rebaselining a MISSING test, we try to rebaseline all png, txt and wav results. However, the actual result may not be the up-to-date result, but the result of the last failure. We should get actual result from the storage only if that result is actually missing. For non-missing result, we should get from the current expectation from local checkout. This is basically the same method as for bug 574272 that we should get the passing result from local checkout and failing result from repository.
,
Jun 14 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a28604ed1b4ebe103318e80f14171ffcc656c2bc commit a28604ed1b4ebe103318e80f14171ffcc656c2bc Author: wangxianzhu <wangxianzhu@chromium.org> Date: Tue Jun 14 20:21:29 2016 Rebaseline the actual missing results only Previously we tried to rebaseline all txt, png and wav results for a test with MISSING actual results. This caused out-dated results were used for non-missing results because the "latest layout test results" storage server contains the latest failure of each test of all recent runs. BUG= 613660 Review-Url: https://codereview.chromium.org/2069743002 Cr-Commit-Position: refs/heads/master@{#399776} [modify] https://crrev.com/a28604ed1b4ebe103318e80f14171ffcc656c2bc/third_party/WebKit/Tools/Scripts/webkitpy/common/net/layouttestresults.py [modify] https://crrev.com/a28604ed1b4ebe103318e80f14171ffcc656c2bc/third_party/WebKit/Tools/Scripts/webkitpy/common/net/layouttestresults_unittest.py [modify] https://crrev.com/a28604ed1b4ebe103318e80f14171ffcc656c2bc/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/models/test_expectations.py [modify] https://crrev.com/a28604ed1b4ebe103318e80f14171ffcc656c2bc/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/views/buildbot_results.py [modify] https://crrev.com/a28604ed1b4ebe103318e80f14171ffcc656c2bc/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py [modify] https://crrev.com/a28604ed1b4ebe103318e80f14171ffcc656c2bc/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_unittest.py [modify] https://crrev.com/a28604ed1b4ebe103318e80f14171ffcc656c2bc/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaselineserver.py
,
Jun 14 2016
,
Jun 15 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a28604ed1b4ebe103318e80f14171ffcc656c2bc commit a28604ed1b4ebe103318e80f14171ffcc656c2bc Author: wangxianzhu <wangxianzhu@chromium.org> Date: Tue Jun 14 20:21:29 2016 Rebaseline the actual missing results only Previously we tried to rebaseline all txt, png and wav results for a test with MISSING actual results. This caused out-dated results were used for non-missing results because the "latest layout test results" storage server contains the latest failure of each test of all recent runs. BUG= 613660 Review-Url: https://codereview.chromium.org/2069743002 Cr-Commit-Position: refs/heads/master@{#399776} [modify] https://crrev.com/a28604ed1b4ebe103318e80f14171ffcc656c2bc/third_party/WebKit/Tools/Scripts/webkitpy/common/net/layouttestresults.py [modify] https://crrev.com/a28604ed1b4ebe103318e80f14171ffcc656c2bc/third_party/WebKit/Tools/Scripts/webkitpy/common/net/layouttestresults_unittest.py [modify] https://crrev.com/a28604ed1b4ebe103318e80f14171ffcc656c2bc/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/models/test_expectations.py [modify] https://crrev.com/a28604ed1b4ebe103318e80f14171ffcc656c2bc/third_party/WebKit/Tools/Scripts/webkitpy/layout_tests/views/buildbot_results.py [modify] https://crrev.com/a28604ed1b4ebe103318e80f14171ffcc656c2bc/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline.py [modify] https://crrev.com/a28604ed1b4ebe103318e80f14171ffcc656c2bc/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaseline_unittest.py [modify] https://crrev.com/a28604ed1b4ebe103318e80f14171ffcc656c2bc/third_party/WebKit/Tools/Scripts/webkitpy/tool/commands/rebaselineserver.py |
||||
►
Sign in to add a comment |
||||
Comment 1 by wangxianzhu@chromium.org
, May 23 2016