Example failed build:

https://build.chromium.org/p/chromium.webkit/builders/WebKit%20Win7/builds/44070

Regressions: Unexpected image-only failures (5)
  fast/repaint/line-flow-with-floats-9.html [ Failure ]
  fast/repaint/nested-fixed-iframe-scrolled.html [ Failure ]
  fast/repaint/repaint-during-scroll-with-zoom.html [ Failure ]
  fast/repaint/resize-scrollable-iframe.html [ Failure ]
  fast/repaint/text-match-document-change.html [ Failure ]

Example output:

https://storage.googleapis.com/chromium-layout-test-archives/WebKit_Win7/44070/layout-test-results.zip

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5285fd021886fa5f78a4dd9e8bde7988ecc12242

commit 5285fd021886fa5f78a4dd9e8bde7988ecc12242
Author: engedy <engedy@chromium.org>
Date: Wed Jul 13 12:26:52 2016

Mark 5 tests in fast/repaint/* as flaky on Win.

fast/repaint/line-flow-with-floats-9.html
fast/repaint/nested-fixed-iframe-scrolled.html
fast/repaint/repaint-during-scroll-with-zoom.html
fast/repaint/resize-scrollable-iframe.html
fast/repaint/text-match-document-change.html

TBR=wangxianzhu@chromium.org
NOTRY=true
BUG= 627798 , 617799 

Review-Url: https://codereview.chromium.org/2140673008
Cr-Commit-Position: refs/heads/master@{#405127}

[modify] https://crrev.com/5285fd021886fa5f78a4dd9e8bde7988ecc12242/third_party/WebKit/LayoutTests/TestExpectations

In the mean time, I have seen the same error in other layout tests too. Same weird blue scrollbar. Have we seen this before?

The blue scrollbars are of the style of layout test. It seems that the tests are flaky about choosing scrollbar scheme.

CC sky@ who might now more about why we are non-deterministically falling back to this theme with the weird blue scrollbars.

s/now/know

The blue scrollbars are correct, not weird actually :)
The baselines on Windows are actually incorrect (which was created when the tests happened to produce the incorrect results).

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5285fd021886fa5f78a4dd9e8bde7988ecc12242

commit 5285fd021886fa5f78a4dd9e8bde7988ecc12242
Author: engedy <engedy@chromium.org>
Date: Wed Jul 13 12:26:52 2016

Mark 5 tests in fast/repaint/* as flaky on Win.

fast/repaint/line-flow-with-floats-9.html
fast/repaint/nested-fixed-iframe-scrolled.html
fast/repaint/repaint-during-scroll-with-zoom.html
fast/repaint/resize-scrollable-iframe.html
fast/repaint/text-match-document-change.html

TBR=wangxianzhu@chromium.org
NOTRY=true
BUG= 627798 , 617799 

Review-Url: https://codereview.chromium.org/2140673008
Cr-Commit-Position: refs/heads/master@{#405127}

[modify] https://crrev.com/5285fd021886fa5f78a4dd9e8bde7988ecc12242/third_party/WebKit/LayoutTests/TestExpectations

These tests are also flaky on Mac and Linux, but because the baselines are correct on these platforms, they are less flaky and normally succeed when retried so they seldom cause webkit_tests step to fail.

http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=fast%2Frepaint%2Fline-flow-with-floats-9.html

 Issue 627902  has been merged into this issue.

It looks like editing/spelling/inline-spelling-markers-hidpi-composited.html on Windows is having the same issue.

http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=editing%2Fspelling%2Finline-spelling-markers-hidpi-composited.html&showExpectations=true

 Issue 627951  has been merged into this issue.

#13: I've just merged  issue 627951 , which was filed separately for editing/spelling/inline-spelling-markers-hidpi-composited.html.

I'm going to also mark it as flaky on Windows, unless someone is already doing it.  It's been keeping one of the site isolation bots mostly red: https://build.chromium.org/p/chromium.fyi/builders/Site%20Isolation%20Win/builds/15400

Please mark it flaky on all platforms: http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=editing%2Fspelling%2Finline-spelling-markers-hidpi-composited.html.

#15: Go ahead. I marked it flaky here https://codereview.chromium.org/2150523003/ but only on Windows 7 because I didn't see it failing on Windows 10.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/87a98e4bdb72d8dd75c89d1c068f26516da0ef6b

commit 87a98e4bdb72d8dd75c89d1c068f26516da0ef6b
Author: alexmos <alexmos@chromium.org>
Date: Thu Jul 14 00:39:14 2016

Mark several layout tests as flaky on all platforms due to incorrect scrollbars.

BUG= 627798 
NOTRY=true
TBR=wangxianzhu@chromium.org

Review-Url: https://codereview.chromium.org/2151723002
Cr-Commit-Position: refs/heads/master@{#405367}

[modify] https://crrev.com/87a98e4bdb72d8dd75c89d1c068f26516da0ef6b/third_party/WebKit/LayoutTests/TestExpectations

It seems that test_runner::TestRunner::use_mock_theme_ is not correctly set, or is not correctly reset for subsequent tests after some test set it to another value. If the value is false (should be false for most of the tests), we will draw the scrollbar in the default theme.

My CL just exposed the issue, but didn't cause the issue.

Correction to #19:
s/should be false for most of the tests/should be true for most of the tests/

 Issue 632007  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6ed36319f28f95540e80cedc13ff0256a620aff4

commit 6ed36319f28f95540e80cedc13ff0256a620aff4
Author: timloh <timloh@chromium.org>
Date: Wed Aug 03 05:40:17 2016

Gardening: Mark some tests as flaky

The following tests are flaky on all platforms, occasionally showing
incorrect scrollbars.
fast/forms/fieldset/fieldset-align.html
fast/forms/select/listbox-clip.html
fast/block/positioning/auto/007.html
compositing/overflow/update-widget-positions-on-nested-frames-and-scrollers.html

BUG= 627798 
NOTRY=TRUE
TBR=wangxianzhu@chromium.org

Review-Url: https://codereview.chromium.org/2202053010
Cr-Commit-Position: refs/heads/master@{#409465}

[modify] https://crrev.com/6ed36319f28f95540e80cedc13ff0256a620aff4/third_party/WebKit/LayoutTests/TestExpectations

I think this may be the same root cause as  https://crbug.com/633707 .

 Issue 635390  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c

commit 002f2e5577c4538c7ef29f8ee7648c14f9bdce4c
Author: Rebaseline Bot <blink-rebaseline-bot@chromium.org>
Date: Tue Aug 09 00:26:15 2016

Auto-rebaseline for r410485

https://chromium.googlesource.com/chromium/src/+/a5191dca2

BUG= 627798 , 633707 
TBR=pdr@chromium.org

Review URL: https://codereview.chromium.org/2225883003 .

Cr-Commit-Position: refs/heads/master@{#410515}

[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/TestExpectations
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/fast/repaint/resize-scrollable-iframe-expected.txt
[delete] https://crrev.com/7b023b1936593c7360fe7a3b6a0d2c6002a174cc/third_party/WebKit/LayoutTests/platform/linux-precise/compositing/overflow/scroll-ancestor-update-expected.png
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/linux/compositing/geometry/video-fixed-scrolling-expected.png
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/linux/compositing/layers-inside-overflow-scroll-expected.png
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/linux/compositing/overflow/update-widget-positions-on-nested-frames-and-scrollers-expected.png
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/linux/compositing/self-painting-layers-expected.png
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/linux/fast/repaint/line-flow-with-floats-9-expected.txt
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/linux/fast/repaint/resize-scrollable-iframe-expected.txt
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/linux/fast/repaint/text-match-document-change-expected.txt
[delete] https://crrev.com/7b023b1936593c7360fe7a3b6a0d2c6002a174cc/third_party/WebKit/LayoutTests/platform/mac-mac10.9/compositing/geometry/video-fixed-scrolling-expected.png
[delete] https://crrev.com/7b023b1936593c7360fe7a3b6a0d2c6002a174cc/third_party/WebKit/LayoutTests/platform/mac-mac10.9/compositing/layers-inside-overflow-scroll-expected.png
[delete] https://crrev.com/7b023b1936593c7360fe7a3b6a0d2c6002a174cc/third_party/WebKit/LayoutTests/platform/mac-mac10.9/compositing/overflow/scroll-ancestor-update-expected.png
[delete] https://crrev.com/7b023b1936593c7360fe7a3b6a0d2c6002a174cc/third_party/WebKit/LayoutTests/platform/mac-mac10.9/compositing/self-painting-layers-expected.png
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/mac/compositing/overflow/update-widget-positions-on-nested-frames-and-scrollers-expected.png
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/mac/fast/repaint/line-flow-with-floats-9-expected.txt
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/mac/fast/repaint/text-match-document-change-expected.txt
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/win/compositing/geometry/video-fixed-scrolling-expected.png
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/win/compositing/layers-inside-overflow-scroll-expected.png
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/win/compositing/overflow/scroll-ancestor-update-expected.png
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/win/compositing/overflow/update-widget-positions-on-nested-frames-and-scrollers-expected.png
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/win/compositing/self-painting-layers-expected.png
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/win/fast/forms/select-popup/popup-menu-appearance-long-expected.png
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/win/fast/forms/select-popup/popup-menu-appearance-many-expected.png
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/win/fast/forms/select-popup/popup-menu-appearance-rtl-default-expected.png
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/win/fast/forms/select-popup/popup-menu-appearance-rtl-expected.png
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/win/fast/forms/select-popup/popup-menu-appearance-zoom090-expected.png
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/win/fast/repaint/line-flow-with-floats-9-expected.txt
[modify] https://crrev.com/002f2e5577c4538c7ef29f8ee7648c14f9bdce4c/third_party/WebKit/LayoutTests/platform/win/fast/repaint/text-match-document-change-expected.txt
[delete] https://crrev.com/7b023b1936593c7360fe7a3b6a0d2c6002a174cc/third_party/WebKit/LayoutTests/platform/win7/compositing/overflow/update-widget-positions-on-nested-frames-and-scrollers-expected.png
[delete] https://crrev.com/7b023b1936593c7360fe7a3b6a0d2c6002a174cc/third_party/WebKit/LayoutTests/platform/win7/editing/spelling/inline-spelling-markers-hidpi-composited-expected.png
[delete] https://crrev.com/7b023b1936593c7360fe7a3b6a0d2c6002a174cc/third_party/WebKit/LayoutTests/platform/win7/fast/forms/select-popup/popup-menu-appearance-long-expected.png
[delete] https://crrev.com/7b023b1936593c7360fe7a3b6a0d2c6002a174cc/third_party/WebKit/LayoutTests/platform/win7/fast/forms/select-popup/popup-menu-appearance-many-expected.png
[delete] https://crrev.com/7b023b1936593c7360fe7a3b6a0d2c6002a174cc/third_party/WebKit/LayoutTests/platform/win7/fast/forms/select-popup/popup-menu-appearance-rtl-default-expected.png
[delete] https://crrev.com/7b023b1936593c7360fe7a3b6a0d2c6002a174cc/third_party/WebKit/LayoutTests/platform/win7/fast/forms/select-popup/popup-menu-appearance-rtl-expected.png
[delete] https://crrev.com/7b023b1936593c7360fe7a3b6a0d2c6002a174cc/third_party/WebKit/LayoutTests/platform/win7/fast/forms/select-popup/popup-menu-appearance-zoom090-expected.png
[delete] https://crrev.com/7b023b1936593c7360fe7a3b6a0d2c6002a174cc/third_party/WebKit/LayoutTests/platform/win7/fast/repaint/line-flow-with-floats-9-expected.png
[delete] https://crrev.com/7b023b1936593c7360fe7a3b6a0d2c6002a174cc/third_party/WebKit/LayoutTests/platform/win7/fast/repaint/nested-fixed-iframe-scrolled-expected.png
[delete] https://crrev.com/7b023b1936593c7360fe7a3b6a0d2c6002a174cc/third_party/WebKit/LayoutTests/platform/win7/fast/repaint/repaint-during-scroll-with-zoom-expected.png
[delete] https://crrev.com/7b023b1936593c7360fe7a3b6a0d2c6002a174cc/third_party/WebKit/LayoutTests/platform/win7/fast/repaint/resize-scrollable-iframe-expected.png
[delete] https://crrev.com/7b023b1936593c7360fe7a3b6a0d2c6002a174cc/third_party/WebKit/LayoutTests/platform/win7/fast/repaint/text-match-document-change-expected.png

 Issue 633026  has been merged into this issue.

Woohoo!