The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/dbf30e19b59f63adc8fd0dafd2e245218bc8d798

commit dbf30e19b59f63adc8fd0dafd2e245218bc8d798
Author: danakj <danakj@chromium.org>
Date: Thu Aug 25 22:51:05 2016

cc: Fix pixeltest expectations for the waterfall.

The waterfall bots have different pixel output from skia scaling than
trybots or local results. Make the fuzzy expectations work for the
waterfall.

TBR=enne
BUG= 606056 , 641056 
NOTRY=true

Review-Url: https://codereview.chromium.org/2286493002
Cr-Commit-Position: refs/heads/master@{#414569}

[modify] https://crrev.com/dbf30e19b59f63adc8fd0dafd2e245218bc8d798/cc/trees/layer_tree_host_pixeltest_scrollbars.cc

Blink-Infra: is this something on your radar by any chance?

Wasn't on my radar before -- cc pixel tests are an area I'm currently unfamiliar with.

Some questions I have: 
 - The cc pixel tests are all part of cc_unittests, right? Not webkit_tests?
 - How do you rebaseline them? (Are the baselines all platform-independent?)

Most release trybots are built w/ dcheck_always_on=true, whereas the release waterfall builders have dcheck_always_on=false, but that should be pretty much the only difference.

Perhaps that might be the cause here as well?

This has been a source of issues in Blink for years, but the alternatives (no dcheck tryserver coverage, or running tests under debug) have always been seen as worse.

I haven't heard of this being an issue for cc/ before, though.

Hm, I was using dcheck_always_on=true locally too, so it's possible..

 - The cc pixel tests are all part of cc_unittests, right? Not webkit_tests?

Correct.

 - How do you rebaseline them? (Are the baselines all platform-independent?)

We just upload a new .png file in cc/test/data/, there's a --cc-rebaseline-pixeltests that will cause any test run with it to replace the .png instead of compare against it.

> Perhaps that might be the cause here as well?

It is in fact DCHECK_ALWAYS_ON. When I remove the fuzzy pixel comparator from the LayerTreeHostScrollbarsPixelTest.HugeTransformScale test, it passes with DCHECKs on, but fails with them off.

> Most release trybots are built w/ dcheck_always_on=true, whereas the release waterfall builders have dcheck_always_on=false

Is this inverted? I rebaselined this test to match what the waterfall does by using its actual output. But this is matching dchecks *on*.

Interesting enough, it looks like some memory gets initialized (or not) differently under DCHECK_IS_ON(). The test had some pixels that end up partially coming from that uninit memory, which msan catches here: https://bugs.chromium.org/p/chromium/issues/detail?id=642011  But it gets (un?)initialized differently depending on DCHECKs.

Thanks for that helpful pointer dpranke!