Fallout from the change to reflection painting?

I investigated, and these are my findings:

The flakiness doesn't appear to have anything to do with either box
reflection or compositing. If I comment out both of those in
compositing/reflections/nested-reflection-animated.html and then rebaseline,
I get the same problem of the test succeeding in a single-test run and
failing in a run of all of compositing/reflections/ (or vice-versa depending
on the recorded baseline)

My guess is that it is a problem to do with flakiness in the runAnimationTest
helper method, where it does not always stop at precisely the same animation
point, and therefore might cause image output flakiness.

Therefore, changing the compoent to animations and requesting feedback on
whether runAnimationTest should be able to exactly reproduce pixel output,
or this is not an achievable property of the system. If not, it seems we need
to turn off pixel output testing of these tests.

runAnimationTest() won't be flaky with the disablePauseAnimationAPI parameter set to false (which is the case in this test). We pause animations at an exact time when using the pause API available on the internals object.

I suspect animations aren't the culprit here given that nested-reflection-anchor-point.html has the same error as nested-reflection-animated.html when run in isolation.

It might be worth converting these tests over to ref tests instead of pixel tests in case that solves the issue. The failure diff I see isn't very meaningful, only the anti-aliased edge of the boxes differ.

 Issue 663571  has been merged into this issue.

The compositing/reflections/nested-reflection-transition.html is a related issue in that the result is not consistent across runs and the reference that is checked in is from the second run.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2e4858e7eef2135034ed003e58a380e0813da9e6

commit 2e4858e7eef2135034ed003e58a380e0813da9e6
Author: schenney <schenney@chromium.org>
Date: Wed Dec 14 20:31:20 2016

Tighten expectations for 3 flaky animated compositing tests.

These 3 tests are flaky when run in random order or even alone.
As a first step in improving them, modify the tests to get away
from setTimeout and to verify that the pauseAnimationAPI is working.

This is unlikely to resolve the flakiness, but it will help narrow
the cause.

R=chrishtr
BUG=653722

Review-Url: https://codereview.chromium.org/2569063002
Cr-Commit-Position: refs/heads/master@{#438608}

[modify] https://crrev.com/2e4858e7eef2135034ed003e58a380e0813da9e6/third_party/WebKit/LayoutTests/compositing/reflections/nested-reflection-anchor-point.html
[modify] https://crrev.com/2e4858e7eef2135034ed003e58a380e0813da9e6/third_party/WebKit/LayoutTests/compositing/reflections/nested-reflection-animated-expected.txt
[modify] https://crrev.com/2e4858e7eef2135034ed003e58a380e0813da9e6/third_party/WebKit/LayoutTests/compositing/reflections/nested-reflection-animated.html
[modify] https://crrev.com/2e4858e7eef2135034ed003e58a380e0813da9e6/third_party/WebKit/LayoutTests/compositing/reflections/nested-reflection-transition-expected.txt
[modify] https://crrev.com/2e4858e7eef2135034ed003e58a380e0813da9e6/third_party/WebKit/LayoutTests/compositing/reflections/nested-reflection-transition.html
[modify] https://crrev.com/2e4858e7eef2135034ed003e58a380e0813da9e6/third_party/WebKit/LayoutTests/platform/linux/compositing/reflections/nested-reflection-animated-expected.png
[modify] https://crrev.com/2e4858e7eef2135034ed003e58a380e0813da9e6/third_party/WebKit/LayoutTests/platform/linux/compositing/reflections/nested-reflection-transition-expected.png
[modify] https://crrev.com/2e4858e7eef2135034ed003e58a380e0813da9e6/third_party/WebKit/LayoutTests/platform/mac/compositing/reflections/nested-reflection-animated-expected.png
[modify] https://crrev.com/2e4858e7eef2135034ed003e58a380e0813da9e6/third_party/WebKit/LayoutTests/platform/mac/compositing/reflections/nested-reflection-transition-expected.png
[modify] https://crrev.com/2e4858e7eef2135034ed003e58a380e0813da9e6/third_party/WebKit/LayoutTests/platform/win/compositing/reflections/nested-reflection-animated-expected.png
[modify] https://crrev.com/2e4858e7eef2135034ed003e58a380e0813da9e6/third_party/WebKit/LayoutTests/platform/win/compositing/reflections/nested-reflection-transition-expected.png

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e9cfdd1cf073da6c905c29d47839b76400e0b08b

commit e9cfdd1cf073da6c905c29d47839b76400e0b08b
Author: ksakamoto <ksakamoto@chromium.org>
Date: Thu Dec 15 04:17:08 2016

Mark compositing/reflections/nested-reflection-transition.html as failing on Win

BUG= 674396 ,653722
TBR=schenney@chromium.org
NOTRY=TRUE

Review-Url: https://codereview.chromium.org/2577893003
Cr-Commit-Position: refs/heads/master@{#438732}

[modify] https://crrev.com/e9cfdd1cf073da6c905c29d47839b76400e0b08b/third_party/WebKit/LayoutTests/TestExpectations

 Issue 674468  has been merged into this issue.

 Issue 674396  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/93ee9b81cc8e0c582c48313b16647eaf3c6afdef

commit 93ee9b81cc8e0c582c48313b16647eaf3c6afdef
Author: schenney <schenney@chromium.org>
Date: Thu Mar 16 19:04:43 2017

Rebaseline various reflection animation results

These tests are probably still flaky on Windows, but
it's hard to tell when they are always failing on Linux.
So rebaseline to get at leat some passing results to
evaluate flakiness.

TBR=chrishtr@chromium.org
BUG=653722

Review-Url: https://codereview.chromium.org/2750913004
Cr-Commit-Position: refs/heads/master@{#457510}

[modify] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/TestExpectations
[modify] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/linux/compositing/reflections/nested-reflection-anchor-point-expected.png
[modify] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/linux/compositing/reflections/nested-reflection-animated-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/linux/compositing/reflections/nested-reflection-transition-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/linux/virtual/disable-spinvalidation/compositing/reflections/nested-reflection-anchor-point-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/linux/virtual/disable-spinvalidation/compositing/reflections/nested-reflection-animated-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/linux/virtual/disable-spinvalidation/compositing/reflections/nested-reflection-transition-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/mac-mac10.10/compositing/reflections/nested-reflection-anchor-point-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/mac-mac10.10/compositing/reflections/nested-reflection-animated-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/mac-mac10.10/virtual/disable-spinvalidation/compositing/reflections/nested-reflection-anchor-point-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/mac-mac10.11/compositing/reflections/nested-reflection-anchor-point-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/mac-mac10.11/compositing/reflections/nested-reflection-animated-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/mac-mac10.11/virtual/disable-spinvalidation/compositing/reflections/nested-reflection-anchor-point-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/mac-mac10.11/virtual/disable-spinvalidation/compositing/reflections/nested-reflection-animated-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/mac-mac10.9/virtual/disable-spinvalidation/compositing/reflections/nested-reflection-anchor-point-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/mac-mac10.9/virtual/disable-spinvalidation/compositing/reflections/nested-reflection-animated-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/mac-retina/compositing/reflections/nested-reflection-animated-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/mac-retina/virtual/disable-spinvalidation/compositing/reflections/nested-reflection-anchor-point-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/mac-retina/virtual/disable-spinvalidation/compositing/reflections/nested-reflection-animated-expected.png
[modify] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/mac/compositing/reflections/nested-reflection-anchor-point-expected.png
[modify] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/mac/compositing/reflections/nested-reflection-animated-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/mac/virtual/disable-spinvalidation/compositing/reflections/nested-reflection-anchor-point-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/mac/virtual/disable-spinvalidation/compositing/reflections/nested-reflection-animated-expected.png
[modify] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/win/compositing/reflections/nested-reflection-anchor-point-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/win/virtual/disable-spinvalidation/compositing/reflections/nested-reflection-anchor-point-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/win/virtual/disable-spinvalidation/compositing/reflections/nested-reflection-animated-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/win7/compositing/reflections/nested-reflection-anchor-point-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/win7/compositing/reflections/nested-reflection-animated-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/win7/compositing/reflections/nested-reflection-transition-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/win7/virtual/disable-spinvalidation/compositing/reflections/nested-reflection-anchor-point-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/win7/virtual/disable-spinvalidation/compositing/reflections/nested-reflection-animated-expected.png
[add] https://crrev.com/93ee9b81cc8e0c582c48313b16647eaf3c6afdef/third_party/WebKit/LayoutTests/platform/win7/virtual/disable-spinvalidation/compositing/reflections/nested-reflection-transition-expected.png