The rebaseline bot thinks it has rebaselined: https://codereview.chromium.org/2651413003/
but in that patch, all it did was remove the TestExpectations lines, and updated issue 35697 ...
So ... I possibly should have specified this bug in the TestExpectations lines? nainar suggests that a mismatch in this bug number could have caused the rebaseline bot to decide it didn't need to do anything.
In desperation, I am going to reland my TestExpectations change with this bug number instead, and see what happens.
I don't think the bug number matters. The rebaseline bot wait until all bots passed the revision which added the NeedsRebaselines, and fetch actual result from the bots. I think this is a bug of the rebaseline-bot.
Yep, I also don't think the bug number matters, and I agree with wangxianhzu that in the short-term this test can be marked [ Win7 ] ... [ Failure ]; for bug 676748 now we have another example to try to reproduce whatever bug there is there :-)
Summary: webkit_tests failing on chromium.webkit/WebKit Win7 and Mac (was: webkit_tests failing on chromium.webkit/WebKit Win7)
I don't know exactly how the rebaseline bot works, but it seems that all the current builders may not be finished yet, since the logs at the bot says it's waiting for a revision.
Docs at https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md#Rebaselining-with-rebaseline_o_matic says:
When all of the continuous builders on the waterfall have cycled, the rebaseline-o-matic bot will commit a patch which includes the new baselines and removes the [ NeedsRebaseline ] entry from TestExpectations.
I'm going to add another CL to rebaseline some of the same tests for Mac.
The below tests are failing on WebKit Win7, going to land a CL marking them as failing in TestExpectations on Win (they were marked as failing on Win10 only previously).
fast/table/backgr_layers-show.html
fast/table/backgr_layers-show-collapsed-border.html
fast/table/backgr_simple-table-row.html
fast/table/backgr_simple-table-row-group-collapsed-border.html
fast/table/backgr_simple-table-cell.html
fast/table/backgr_position-table-cell.html
This isn't the first time using:
crbug.com/bugnumber [ Mac Win ] test/name.html [ NeedsRebaseline ]
has unleashed mayhem. Should I just avoid [ Mac Win ] in future when I rebaseline and do this instead?
crbug.com/bugnumber test/name.html [ NeedsRebaseline ]
I think that even if only one or two platforms need new baselines, adding a NeedsRebaseline line with no OS specifiers should always work, so if you need to rebaseline via rebaseline-o-matic in the future, there's no reason not to just omit the OS specifier part.
However, in the future when you need to rebaseline you can also get new baselines from try jobs by running `webkit-patch rebaseline-cl` (https://chromium.googlesource.com/chromium/src/+/master/docs/testing/layout_test_expectations.md#How-to-rebaseline). The advantage of this way is that you can run try jobs again before committing, thus avoiding mayhem like this :-)
I find I get a lot of bogus results using rebaseline-cl, especially with whatever happens to be flaky that day on Windows/Mac. I will persist with it in future though. I definitely don't like to suck up so much of everyone's time with this stuff.
Makes sense - I plan to keep trying to make improvements to rebaseline-cl, so hopefully that can be made more reliable in avoiding bogus results. As a quick note for next time you try it, you can also directly pass the set of tests to rebaseline as args to the script, or pass --only-changed-tests if you want it to ignore any tests that weren't changed in the current CL.
Current status of this as of 2017-01-28:
Failing with minor pixel baseline mismatch, but layout/positioning still looks OK to me:
fast/text/ellipsis-with-list-marker-in-rtl-flow.html is failing on Mac 10.11
fast/text/ellipsis-with-list-marker-in-ltr-flow.html is failing on Win7
Failing with test console message mismatch (due to non-deterministic ordering):
http/tests/security/xss-DENIED-window-name-navigator.html on Win7
Comment 1 by suzyh@chromium.org
, Jan 26 2017