https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.gpu.fyi%2FMac_Experimental_Retina_Release__AMD_%2F2574%2F%2B%2Frecipes%2Fsteps%2Fpixel_test%2F0%2Fstdout

https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.gpu.fyi%2FMac_Experimental_Retina_Release__AMD_%2F2576%2F%2B%2Frecipes%2Fsteps%2Fscreenshot_sync_tests%2F0%2Fstdout

In the linked stdout for screenshot_sync_tests, there are 8 individual failures. All of the actual color values are within about 1-3 of the expected value in each channel, so it doesn't really sound like a window blocking the test area.

Wonder if this could be new in macOS 10.13?

A few recent failures can be seen more easily here:
https://build.chromium.org/p/chromium.gpu.fyi/buildslaves/build52-b1

I have no idea what's going on here, but these are large test groups and I don't want to disable them.

Actually just 4 individual failures:
expected (242, 225, 182), got (241, 224, 184)
expected (185, 219,  97), got (186, 217, 105)
expected (189,  82, 126), got (187,  84, 126)
expected (133,  36,  96), got (132,  39,  95)

ccameron: do you think there could be a subtle color management issue here? I don't know why these tests would flake (across a single machine) but the values seem to indicate something along those lines.

It's very strange that these would flake.

It could be that the system settings change is racing with the test running (it does take a split-second for the color profile to take effect).

A simple fix would be to just force an sRGB profile and not restore it on exit. That is somewhat hostile to users who run this locally, though (they'd have to re-set their profile in SystemSettings).

Thanks for your feedback Chris. We can easily add a command-line flag for these tests, produced by generate_buildbot_json.py, which would leave the color profile enabled after test exit. That way only the bots would be affected.

It seems to me that if there's a race then we'll still need to try to resolve it by pausing a bit after setting the color profile upon starting the tests.

> It seems to me that if there's a race then we'll still need to try
> to resolve it by pausing a bit after setting the color profile upon
> starting the tests.

Looking at the output from pixel_tests in #1, it appears that every test failed -- that's a good ~minute that the color profile failed to apply. So, I suspect a delay won't take effect.

I'm still trying to see if this repros locally (no luck yet). I'll poke at generate_buildbot_json to see if I can figure out the magic there.

There's still the possibility that this is some kind of precision issue, although it seems less likely; I don't know where they could be creeping in.

I'll assign to you for now, feel free to assign back if needed.

Some of the differences in the blues are pretty big, like:

 > expected (185, 219,  97), got (186, 217, 105)

So I suspect it's not a precision issue.

See comments on https://chromium-review.googlesource.com/770131 about apparent race conditions in restoring the system's color profile.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3788054defbfcc7da051ae9f136dad58b99046f0

commit 3788054defbfcc7da051ae9f136dad58b99046f0
Author: Christopher Cameron <ccameron@chromium.org>
Date: Fri Nov 17 11:07:51 2017

Add --ensure-force-color-profile flag for pixel tests

This crashes the browser if the color profile that was specified at the
command line does not match the color profile on the display.

This is desirable for pixel tests because the pixel test output is
subject to the operating system's color conversion, and will not be
predictable or useful ness the operating system's color space is
controlled for.

Bug:  784456 
Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: I15372cba6291bc764fe862ff6112a9f41d3ce929
Reviewed-on: https://chromium-review.googlesource.com/770131
Commit-Queue: ccameron <ccameron@chromium.org>
Reviewed-by: Kai Ninomiya <kainino@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#517370}
[modify] https://crrev.com/3788054defbfcc7da051ae9f136dad58b99046f0/content/test/gpu/gpu_tests/maps_integration_test.py
[modify] https://crrev.com/3788054defbfcc7da051ae9f136dad58b99046f0/content/test/gpu/gpu_tests/pixel_integration_test.py
[modify] https://crrev.com/3788054defbfcc7da051ae9f136dad58b99046f0/content/test/gpu/gpu_tests/screenshot_sync_integration_test.py
[modify] https://crrev.com/3788054defbfcc7da051ae9f136dad58b99046f0/ui/display/display.cc
[modify] https://crrev.com/3788054defbfcc7da051ae9f136dad58b99046f0/ui/display/display.h
[modify] https://crrev.com/3788054defbfcc7da051ae9f136dad58b99046f0/ui/display/display_switches.cc
[modify] https://crrev.com/3788054defbfcc7da051ae9f136dad58b99046f0/ui/display/display_switches.h
[modify] https://crrev.com/3788054defbfcc7da051ae9f136dad58b99046f0/ui/display/mac/screen_mac.mm

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/19ac06a575adbaea59e385a5ee6af79aa5daf0bd

commit 19ac06a575adbaea59e385a5ee6af79aa5daf0bd
Author: Kenneth Russell <kbr@chromium.org>
Date: Fri Nov 17 23:36:51 2017

Add dont-restore-color-profile-after-test command line flag.

There seem to be OS-level race conditions in restoring the system's
color profile (to Color LCD from the sRGB color profile) and these
seem to be worked around by leaving the system using the sRGB profile
after the tests exit.

BUG= 784456 

Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Change-Id: I09e588f1e8f6dc9c948fd6f2a30e912bc88c820e
Reviewed-on: https://chromium-review.googlesource.com/775625
Reviewed-by: ccameron <ccameron@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#517608}
[modify] https://crrev.com/19ac06a575adbaea59e385a5ee6af79aa5daf0bd/content/test/gpu/generate_buildbot_json.py
[modify] https://crrev.com/19ac06a575adbaea59e385a5ee6af79aa5daf0bd/content/test/gpu/gpu_tests/cloud_storage_integration_test_base.py
[modify] https://crrev.com/19ac06a575adbaea59e385a5ee6af79aa5daf0bd/content/test/gpu/gpu_tests/color_profile_manager.py
[modify] https://crrev.com/19ac06a575adbaea59e385a5ee6af79aa5daf0bd/content/test/gpu/gpu_tests/maps_integration_test.py
[modify] https://crrev.com/19ac06a575adbaea59e385a5ee6af79aa5daf0bd/content/test/gpu/gpu_tests/pixel_integration_test.py
[modify] https://crrev.com/19ac06a575adbaea59e385a5ee6af79aa5daf0bd/content/test/gpu/gpu_tests/screenshot_sync_integration_test.py
[modify] https://crrev.com/19ac06a575adbaea59e385a5ee6af79aa5daf0bd/testing/buildbot/chromium.gpu.fyi.json
[modify] https://crrev.com/19ac06a575adbaea59e385a5ee6af79aa5daf0bd/testing/buildbot/chromium.gpu.json
[modify] https://crrev.com/19ac06a575adbaea59e385a5ee6af79aa5daf0bd/testing/buildbot/client.v8.fyi.json

Let's monitor this bot for a little while and see if it's stabilized. If so, we can finally close this.

That seems to have done it. The attached screenshot shows this bot:
https://ci.chromium.org/buildbot/chromium.gpu.fyi/Mac%20Experimental%20Retina%20Release%20%28AMD%29/?limit=200

going reliably green after the last CL landed.

Thanks Chris for your help on this.

Deleted: Screen Shot 2017-11-20 at 11.22.47 AM.png 1.1 MB

	Screen Shot 2017-11-20 at 11.22.47 AM.png 1.1 MB View Download

Sorry for the churn on this. It looks like the CHECK added does indeed trigger, so this should be easy to diagnose if it ever comes up again.

INFO:root:Browser target: /devtools/browser/c0e46c3c-150a-487a-9c0d-15b5f0ce5075
[9017:775:1117/071422.181315:FATAL:screen_mac.mm(98)] Check failed: screen_color_space == display.color_space() ({primaries:[[0.4443,0.3794,0.1404,],[0.2248,0.7262,0.0491,],[0.0055,0.0780,0.7415,],], transfer:0.0774*x + 0.0000 if x < 0.0404 else (0.9479*x + 0.0521)**2.4000 + 0.0000, matrix:RGB, range:FULL} vs. {primaries:BT709, transfer:IEC61966_2_1, matrix:RGB, range:FULL})The display's color space does not match the color space that was forced by the command line. This will cause pixel tests to fail.
0   Chromium Framework                  0x00000001059bfa3c base::debug::StackTrace::StackTrace(unsigned long) + 28
1   Chromium Framework                  0x00000001059e47b2 logging::LogMessage::~LogMessage() + 210
2   Chromium Framework                  0x00000001068f3e4c display::(anonymous namespace)::BuildDisplayForScreen(NSScreen*) + 860

Awesome, really glad to have that improved diagnostic in there. Without it, it nearly didn't occur to me that it might be a color management issue so I could +you :)

Issue 779710 has been merged into this issue.