Looks like the Skia roll again...msarett's color space CL relanded and re-rolled.

I'm not sure why skia-deps-roller's CL didn't detect this, though. The changes look to be non-trivial, so maybe we can just rebaseline, but it would be better if these could have been suppressed with [ NeedsManualRebaseline ] before the Skia change landed.

Looks like we might be able to rebaseline in this case. I have skia-deps-roller suspended for the moment, in case we do end up needing to revert the roll anyhow. Will resume it if this works.

Yes this one is on me.

I wrote an experimental color xform implementation (only enabled on Linux debug).  That's how it got past the CQ.  But now debug bots are failing - I should have seen this coming.

I guess it's not possible to have different test expectations for Linux debug and Linux release?  So I don't think a rebaseline will work.  I think I need to turn this off - will write a Skia CL.

OK. In the meantime I'm going to roll Skia back, and we can let the fix roll back in afterwards.

Skia unroll is https://codereview.chromium.org/2547023003, in CQ

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/042057cf2558a1c365309e42c690f327cdbe67c8

commit 042057cf2558a1c365309e42c690f327cdbe67c8
Author: jbroman <jbroman@chromium.org>
Date: Fri Dec 02 15:07:37 2016

Revert Skia to 97aadfce6.

This reverts three Skia rolls, to get back to before this change:
2016-12-01 msarett SkColorSpaceXform bug fixes attempt 2

Which seems to be responsible for ~100 layout test failures (due to
small rasterization differences) on Linux debug.

BUG= 670620 
TBR=msarett@chromium.org,brianosman@chromium.org
NOTRY=true

Review-Url: https://codereview.chromium.org/2547023003
Cr-Commit-Position: refs/heads/master@{#435935}

[modify] https://crrev.com/042057cf2558a1c365309e42c690f327cdbe67c8/DEPS

To be more complete here, this fails on linux_chromium_rel_ng but passed the CQ, because:
- linux_chromium_rel_ng has dcheck_always_on=true, which sets SK_DEBUG (so Skia treats it as a debug build, even though it's a "release" bot)
- the skia-deps-roller CL does not trigger webkit_tests on linux_chromium_rel_ng, because DEPS changes don't trigger webkit_tests (only changes inside third_party/WebKit do)
- linux_trusty_blink_rel, which skia-deps-roller explicitly triggers to get Blink coverage, is a release build without dchecks on (and so SK_DEBUG wasn't set, and so the trybot passed the roll CL)

In future I'd advise against pixel output-affecting features being triggered by debug/no-debug in Chromium. There might also be an action item here to find a way to get better coverage of this configuration in the CQ for skia rolls.

(cc skia sheriff, fyi)

This must be from using FMA in matrix_3x4, store_8888, etc. right?

Seems like we should enable for all Linux bots and rebaseline, or for now only enable on some platform that has no layout tests (are there any?).

If you control the enabling from the Chromium side (I think this is what SkUserConfig.h is for?), then you can land rebaselines in the same CL as the enabling change, as well as making it possible to revert the enabling this feature without having to revert the entire Skia roll.

Doing "all Linux" sounds more supportable to me than "just debug linux". I think we might not have Android layout test bots at the moment, but I wouldn't rely on that fact.

> This must be from using FMA in matrix_3x4, store_8888, etc. right?

Sure, and there might be a few other possibilities.  The diffs are minimal and fine.

> Seems like we should enable for all Linux bots and rebaseline, or for now only
> enable on some platform that has no layout tests (are there any?).

I've gone ahead and turned everything off for now.  Then we can reenable (or not) behind a Chrome flag as #13 suggests.
https://skia-review.googlesource.com/c/5501/
https://codereview.chromium.org/2543833004/

The following revision refers to this bug:
  https://skia.googlesource.com/skia.git/+/d459f3c15adda1544e64a0d5d15bc7970635debd

commit d459f3c15adda1544e64a0d5d15bc7970635debd
Author: Matt Sarett <msarett@google.com>
Date: Fri Dec 02 15:01:48 2016

Fix Chrome Linux - temporarily turn off xform pipeline

BUG:670620

Change-Id: Ic481d09a7112ef05f53fa1f94a7c155870c43408
Reviewed-on: https://skia-review.googlesource.com/5501
Commit-Queue: Matt Sarett <msarett@google.com>
Reviewed-by: Brian Osman <brianosman@google.com>
Reviewed-by: Jeremy Roman <jbroman@chromium.org>

[modify] https://crrev.com/d459f3c15adda1544e64a0d5d15bc7970635debd/src/core/SkColorSpaceXform.cpp

I've re-enabled the roller, offending change is now disabled, and will remain behind a flag in the short term.

Thanks everyone for the quick investigation and action. Should we mark this Fixed? It does seem that there are some follow-up actions to take - considering adding dcheck_always_on=true to linux_trusty_blink_rel, and perhaps other things.

We can't add dcheck_always_on to linux_trusty_blink_rel, as it would then produce failures that wouldn't show up on the waterfall builders, and that would mess up rebaselining.

We could add a new builder to capture the release+dcheck configuration, and run it on rolls. Or you could run the tests on a debug builder on rolls.

Since the breakage is resolved, I'm going to mark this as fixed.