kbr@ Can you look into this or route it to the correct person?

Hmm. GpuRasterization.BlueBox is working fine; it's GpuRasterization.ConcavePaths that's failing.

Were any changes made in this area (handling of concave paths with Ganesh) recently, either in Skia or Chromium? Stephen, can you help track down a potential culprit?

Note: it's unfortunate that a screen shot of the failing test isn't uploaded to cloud storage when the test fails. That would probably speed up diagnosis. It should be fairly easy to add this to the test.

Can you disable the test so it's not causing flake on the CQ? I'm not sure how to disable this python based GPU test thing.

Yes. https://codereview.chromium.org/1818293002/ will suppress the failures.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6e1e7324d2e095d6b4d71c62646c9ffff61113b8

commit 6e1e7324d2e095d6b4d71c62646c9ffff61113b8
Author: kbr <kbr@chromium.org>
Date: Tue Mar 22 03:11:48 2016

Marked GpuRasterization.ConcavePaths flaky on Mac.

BUG= 596730 
TBR=esprehn@chromium.org, bajones@chromium.org, senorblanco@chromium.org
NOTRY=true

Review URL: https://codereview.chromium.org/1818293002

Cr-Commit-Position: refs/heads/master@{#382492}

[modify] https://crrev.com/6e1e7324d2e095d6b4d71c62646c9ffff61113b8/content/test/gpu/gpu_tests/gpu_rasterization_expectations.py

Looking at the Skia change log I don't see anything that jumps out as being related.

The only thing that pops to mind is that Eric recently changed the way the MSAA blacklist works. I haven't checked if that's in the range, though. Is this only showing up on the trybots, or is it on the main waterfall too?

On this bot:
https://build.chromium.org/p/chromium.gpu/builders/Mac%20Retina%20Release?numbuilds=200

the first failure I see is from March 19 (3 days ago):
https://build.chromium.org/p/chromium.gpu/builders/Mac%20Retina%20Release/builds/50518

My suppression didn't help. Here's a failure demonstrating that:
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/198897

Here's the log from the tryserver which shows that the failure mode looks like the window is all black, and that happens repeatedly, even when retrying the test:

[ RUN      ] GpuRasterization.ConcavePaths
[10100:1287:0322/101058:INFO:CONSOLE(0)] "'webkitRequestAnimationFrame' is vendor-specific. Please use the standard 'requestAnimationFrame' instead.", source:  (0)
[10100:1287:0322/101105:INFO:CONSOLE(0)] "'webkitRequestAnimationFrame' is vendor-specific. Please use the standard 'requestAnimationFrame' instead.", source:  (0)
[10100:1287:0322/101112:INFO:CONSOLE(0)] "'webkitRequestAnimationFrame' is vendor-specific. Please use the standard 'requestAnimationFrame' instead.", source:  (0)
FLAKY TEST FAILURE, retrying: GpuRasterization.ConcavePaths
FLAKY TEST FAILURE, retrying: GpuRasterization.ConcavePaths
Traceback (most recent call last):
  File "/private/var/folders/rs/_swg7nyj5pl69vk6j2j8zv700000gm/T/runS4rn0h/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 84, in _RunStoryAndProcessErrorIfNeeded
    state.RunStory(results)
  File "/private/var/folders/rs/_swg7nyj5pl69vk6j2j8zv700000gm/T/runS4rn0h/content/test/gpu/gpu_tests/gpu_test_base.py", line 111, in RunStory
    RunStoryWithRetries(GpuSharedPageState, self, results)
  File "/private/var/folders/rs/_swg7nyj5pl69vk6j2j8zv700000gm/T/runS4rn0h/content/test/gpu/gpu_tests/gpu_test_base.py", line 95, in RunStoryWithRetries
    super(cls, shared_page_state).RunStory(results)
  File "/private/var/folders/rs/_swg7nyj5pl69vk6j2j8zv700000gm/T/runS4rn0h/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 331, in RunStory
    self._current_page, self._current_tab, results)
  File "/private/var/folders/rs/_swg7nyj5pl69vk6j2j8zv700000gm/T/runS4rn0h/content/test/gpu/gpu_tests/gpu_rasterization.py", line 67, in ValidateAndMeasurePage
    device_pixel_ratio)
  File "/private/var/folders/rs/_swg7nyj5pl69vk6j2j8zv700000gm/T/runS4rn0h/content/test/gpu/gpu_tests/cloud_storage_test_base.py", line 222, in _ValidateScreenshotSamples
    _CompareScreenshotSamples(screenshot, expectations, device_pixel_ratio)
  File "/private/var/folders/rs/_swg7nyj5pl69vk6j2j8zv700000gm/T/runS4rn0h/content/test/gpu/gpu_tests/cloud_storage_test_base.py", line 55, in _CompareScreenshotSamples
    str(actual_color.b) + "]")
Failure: Expected pixel at [20, 50] to be [255, 215, 0] but got [0, 0, 0]

[  FAILED  ] GpuRasterization.ConcavePaths (21835 ms)
[  PASSED  ] 1 test.
[  FAILED  ] 1 test, listed below:
[  FAILED  ]  GpuRasterization.ConcavePaths

The timing is pretty coincidental with my MSAA blacklist patch (first flakes started a few hours after that went in).... Seems odd that this would cause flakiness as opposed to failures, but I'll look into this a bit.

I had missed adding an IPC struct trait to the previous CL - might explain the flakiness if this led to the variable being uninitialized. I have a patch up for that crrev.com/1826673002 - hopefully this will make the issue go away or become 100% reproducible. Tried to repro on my local mac nVidia machine, but didn't hit the issue.

Detected 9 new flakes for test/step "gpu_rasterization_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiRncHVfcmFzdGVyaXphdGlvbl90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Thanks Eric for continuing to investigate this. Let me please assign it to you. Note that the previous marking of the test as flaky didn't work -- the test fails 3 times in a row when it fails at all.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9a7b8ea49fe9b5d8509a9a43275809ab759853a8

commit 9a7b8ea49fe9b5d8509a9a43275809ab759853a8
Author: ericrk <ericrk@chromium.org>
Date: Wed Mar 23 18:03:06 2016

Add missing IPC struct member msaa_is_slow.

Missed this in previous CL: crrev.com/1781093002 - added this field to the capabilities, but not to the IPC struct traits.

Might explain some of the flakiness in GPU bots.

BUG= 596730 ,  596475 
CQ_INCLUDE_TRYBOTS=tryserver.chromium.win:win_optional_gpu_tests_rel

Review URL: https://codereview.chromium.org/1826673002

Cr-Commit-Position: refs/heads/master@{#382879}

[modify] https://crrev.com/9a7b8ea49fe9b5d8509a9a43275809ab759853a8/gpu/ipc/common/gpu_command_buffer_traits_multi.h

I've landed the CL which *may* address the issue. I will watch for future flakes and see what impact my CL has. Thanks for the info regarding flakiness.

No flakes for the last 24 hours... looking promising, however I still want to understand why my patch would have caused this. At the end of the day, all my patch should have done is modify the number of MSAA samples used by CC/Skia. The bug in my patch just made it non-deterministic how many samples we'd use (the correct value or 0). I've tried both these options locally on my Nvidia mac and haven't seen a flake.

It is odd that the failures wouldn't be reproducible with the flag set either to true or false. Since the memory was uninitialized maybe it was being set to some other value and there was some comparison like (val == true) which was failing? No good guesses.

Would you mind removing the suppression added in https://codereview.chromium.org/1818293002 and watching the bots?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6c260f8ac6796357d31be90f3a713bb24a21acb4

commit 6c260f8ac6796357d31be90f3a713bb24a21acb4
Author: ericrk <ericrk@chromium.org>
Date: Thu Mar 24 22:25:33 2016

Unmark GpuRasterization.ConcavePaths flaky on mac

With crrev.com/1826673002, this should no longer be flaky.
Reverting the expectation added in crrev.com/1818293002.

BUG= 596730 

Review URL: https://codereview.chromium.org/1827203002

Cr-Commit-Position: refs/heads/master@{#383164}

[modify] https://crrev.com/6c260f8ac6796357d31be90f3a713bb24a21acb4/content/test/gpu/gpu_tests/gpu_rasterization_expectations.py

No new flakes - looks like my patch removed the issue. Going to drop priority, but will leave this open as I'd like to investigate further.

Very good Eric. Thanks for tracking this down and following up.

Detected 3 new flakes for test/step "gpu_rasterization_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiRncHVfcmFzdGVyaXphdGlvbl90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Hmmm... the last 3 flakes were all "purple" infra issues that I've been tracking in  issue 599911 .  I don't think this is a reoccurence of flakiness.  Based on this - I'll remove this from the sheriffs' queue - please shout at me if you think this is wrong.

Detected 4 new flakes for test/step "gpu_rasterization_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiRncHVfcmFzdGVyaXphdGlvbl90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Removing Sheriff-Chromium label to remove this from the sheriff queue, since this is assigned.

Detected 3 new flakes for test/step "gpu_rasterization_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiRncHVfcmFzdGVyaXphdGlvbl90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Detected 3 new flakes for test/step "gpu_rasterization_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiRncHVfcmFzdGVyaXphdGlvbl90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Same reason as #27.

The large number of purple steps are  Issue 550016 , which has been mitigated at this point.

In order to prevent chromium-try-flakes from incorrectly posting more updates on this bug I'm calling it fixed. Eric, please file a new bug if you're going to track down why the uninitialized msaa_is_slow flag caused flakes. Thanks.