The last 4 flakes were all "purple" infra issues.

The last 4 flakes are on win_chromium_rel_ng and are not purple, but are proper red.

Seems like a timeout in ScreenshotSync.WithCanvas

ScreenshotSync.WithCanvas is also flaking on the waterfall, on ATI GPU on Windows on Windows-2008ServerR2-SP1. e.g: https://build.chromium.org/p/chromium.gpu/builders/Win7%20Release%20%28ATI%29/builds/14475

The log is:
TimeoutException: 
********************************************************************************
(c:\users\chrome~1\appdata\local\temp\run5wxyuq\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py:315 _ConvertExceptionFromInspectorWebsocket) The app is probably crashed:

Stack Trace:
********************************************************************************
	No crash dump found.
********************************************************************************
Standard output:
********************************************************************************
********************************************************************************
********************************************************************************
(c:\users\chrome~1\appdata\local\temp\run5wxyuq\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py:320 _ConvertExceptionFromInspectorWebsocket) Original exception:
timed out
********************************************************************************
(c:\users\chrome~1\appdata\local\temp\run5wxyuq\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py:341 _AddDebuggingInformation) Received a socket error in the browser connection and the tab still exists. The operation probably timed out.
********************************************************************************
(c:\users\chrome~1\appdata\local\temp\run5wxyuq\third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py:342 _AddDebuggingInformation) Debugger url: ws://127.0.0.1:49424/devtools/page/EE4D6A9A-522E-46E5-B118-A96D2D7942F2

Ali, can you please build locally and see if you can reproduce this and get any more information than the bots are providing? It is certainly caused by some regression elsewhere in Chromium.

Any suggestions on how to disable this test while it's being deflaked?

https://code.google.com/p/chromium/codesearch#chromium/src/content/test/gpu/gpu_tests/screenshot_sync.py&q=ScreenshotSync.WithCanvas&sq=package:chromium&type=cs&l=85

Just comment out the call to AddStory and add a link to this bug?

I'm sorry I haven't been able to track this down yet. I've had to re-image and re-setup my Windows box, and am still in the process of doing that. Also, my Windows box has an Nvidia GPU and all the flakes are on ATI only, so I'm not sure if I'll be able to reproduce locally.

@6, I believe that disabling the test involves adding to screenshot_sync_expectations.py, similar to the other *_expectations.py in that directory.

Please see https://sites.google.com/a/chromium.org/dev/developers/tree-sheriffs/sheriff-details-chromium?pli=1#TOC-Disabling-WebGL-conformance-tests-or-other-GPU-tests: for instructions on marking tests flaky, etc.

Removing Sheriff-Chromium label to remove this from the sheriff queue, since this is assigned.

I wasn't able to repro this on my Windows box (that has an Nvidia GPU); I ran this test with 1000 repeats and it passed every time.

Passing on to this week's pixel wrangler. bajones@, if you happen to have access to a Windows machine with an ATI GPU, perhaps you'll have more luck with this than I did.

Taking over this bug.

 Issue 602313  has been merged into this issue.

As noted in  Issue 602313 , flakiness in these tests is showing up on Mac OS too.

Suppressing these failures in https://codereview.chromium.org/1919623002 . Downgrading.
I couldn't easily reproduce the failures in  Issue 605699 .

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c8d7fb1437ad351bae551f5a7aaaf736e83a07ac

commit c8d7fb1437ad351bae551f5a7aaaf736e83a07ac
Author: kbr <kbr@chromium.org>
Date: Sat Apr 23 03:41:53 2016

Suppress screenshot_sync_tests flakiness on Mac with Intel GPU.

Update the bug ID for the preexisting Windows suppression.

BUG= 599776 ,  605699 
CQ_INCLUDE_TRYBOTS=tryserver.chromium.linux:linux_optional_gpu_tests_rel;tryserver.chromium.mac:mac_optional_gpu_tests_rel;tryserver.chromium.win:win_optional_gpu_tests_rel
NOTRY=true
TBR=zmo@chromium.org

Review URL: https://codereview.chromium.org/1919623002

Cr-Commit-Position: refs/heads/master@{#389356}

[modify] https://crrev.com/c8d7fb1437ad351bae551f5a7aaaf736e83a07ac/content/test/gpu/gpu_tests/screenshot_sync_expectations.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c8d7fb1437ad351bae551f5a7aaaf736e83a07ac

commit c8d7fb1437ad351bae551f5a7aaaf736e83a07ac
Author: kbr <kbr@chromium.org>
Date: Sat Apr 23 03:41:53 2016

Suppress screenshot_sync_tests flakiness on Mac with Intel GPU.

Update the bug ID for the preexisting Windows suppression.

BUG= 599776 ,  605699 
CQ_INCLUDE_TRYBOTS=tryserver.chromium.linux:linux_optional_gpu_tests_rel;tryserver.chromium.mac:mac_optional_gpu_tests_rel;tryserver.chromium.win:win_optional_gpu_tests_rel
NOTRY=true
TBR=zmo@chromium.org

Review URL: https://codereview.chromium.org/1919623002

Cr-Commit-Position: refs/heads/master@{#389356}

[modify] https://crrev.com/c8d7fb1437ad351bae551f5a7aaaf736e83a07ac/content/test/gpu/gpu_tests/screenshot_sync_expectations.py

This is now also occuring on mac amd
https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Retina%20Debug%20%28AMD%29/builds/8718
https://build.chromium.org/p/chromium.gpu/builders/Mac%2010.10%20Retina%20Debug%20%28AMD%29/builds/8695

The backtrace indicates a DevTools timeout while capturing the screenshot. This is affecting other GPU tests on the CQ which capture pixels.

Traceback (most recent call last):
  File "/b/swarm_slave/work/isolated/runYiQvC5/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 84, in _RunStoryAndProcessErrorIfNeeded
    state.RunStory(results)
  File "/b/swarm_slave/work/isolated/runYiQvC5/content/test/gpu/gpu_tests/gpu_test_base.py", line 111, in RunStory
    RunStoryWithRetries(GpuSharedPageState, self, results)
  File "/b/swarm_slave/work/isolated/runYiQvC5/content/test/gpu/gpu_tests/gpu_test_base.py", line 72, in RunStoryWithRetries
    super(cls, shared_page_state).RunStory(results)
  File "/b/swarm_slave/work/isolated/runYiQvC5/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 308, in RunStory
    self._current_page, self._current_tab, results)
  File "/b/swarm_slave/work/isolated/runYiQvC5/content/test/gpu/gpu_tests/screenshot_sync.py", line 57, in ValidateAndMeasurePage
    CheckScreenshot()
  File "/b/swarm_slave/work/isolated/runYiQvC5/content/test/gpu/gpu_tests/screenshot_sync.py", line 46, in CheckScreenshot
    screenshot = tab.Screenshot(5)
  File "/b/swarm_slave/work/isolated/runYiQvC5/third_party/catapult/telemetry/telemetry/internal/browser/tab.py", line 117, in Screenshot
    return self._inspector_backend.Screenshot(timeout)
  File "/b/swarm_slave/work/isolated/runYiQvC5/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 35, in inner
    inspector_backend._ConvertExceptionFromInspectorWebsocket(e)
  File "/b/swarm_slave/work/isolated/runYiQvC5/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 32, in inner
    return func(inspector_backend, *args, **kwargs)
  File "/b/swarm_slave/work/isolated/runYiQvC5/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 139, in Screenshot
    return self._page.CaptureScreenshot(timeout)
  File "/b/swarm_slave/work/isolated/runYiQvC5/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_page.py", line 146, in CaptureScreenshot
    res = self._inspector_websocket.SyncRequest(request, timeout)
  File "/b/swarm_slave/work/isolated/runYiQvC5/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 110, in SyncRequest
    res = self._Receive(timeout)
  File "/b/swarm_slave/work/isolated/runYiQvC5/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 149, in _Receive
    data = self._socket.recv()
  File "/b/swarm_slave/work/isolated/runYiQvC5/third_party/catapult/telemetry/third_party/websocket-client/websocket.py", line 596, in recv
    opcode, data = self.recv_data()
  File "/b/swarm_slave/work/isolated/runYiQvC5/third_party/catapult/telemetry/third_party/websocket-client/websocket.py", line 606, in recv_data
    frame = self.recv_frame()
  File "/b/swarm_slave/work/isolated/runYiQvC5/third_party/catapult/telemetry/third_party/websocket-client/websocket.py", line 637, in recv_frame
    self._frame_header = self._recv_strict(2)
  File "/b/swarm_slave/work/isolated/runYiQvC5/third_party/catapult/telemetry/third_party/websocket-client/websocket.py", line 746, in _recv_strict
    bytes = self._recv(shortage)
  File "/b/swarm_slave/work/isolated/runYiQvC5/third_party/catapult/telemetry/third_party/websocket-client/websocket.py", line 732, in _recv
    raise WebSocketTimeoutException(e.message)


Pavel, can we work together to try to reproduce and debug this? If we can't reproduce, can we add logging to the bots to track down what's happening?

caseq@ and I are investigating this on the affected hardware (Mac Retina with AMD GPU.)

We tracked this down to a power-management issue -- when display goes to sleep, apparently so does the GPU and a call to mac version of GrabViewSnapshot() hangs around https://cs.chromium.org/chromium/src/ui/snapshot/snapshot_mac.mm?rcl=0&l=58:

  NSData* data = [rep representationUsingType:NSPNGFileType properties:@{}];

... till the GPU is awaken.

Note this behavior seems to be different on older NVidia-based MBPs, where a call just returns black pixel instead.

A way to reproduce this locally is to bump tab.Screenshot() timeout here:
https://cs.chromium.org/chromium/src/content/test/gpu/gpu_tests/screenshot_sync.py?rcl=0&l=46

then run the test while using hot corner to force display to turn off.
This might be addressed with PowerSaveBlocker (or corresponding extension API), but configuring bots is probably more practical.

Based on offline discussion we decided to use PowerSaveBlocker while there are outstanding screenshot requests and expose an explicit method to manager power saving state via Page domain of devtools protocol.

 Issue 614394  has been merged into this issue.

 Issue 615548  has been merged into this issue.

 Issue 616629  has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/2d6574bd49f71c153f24f0e78671fbcb481a2aaa

commit 2d6574bd49f71c153f24f0e78671fbcb481a2aaa
Author: caseq <caseq@chromium.org>
Date: Tue Jun 07 08:07:56 2016

Prevent display going to sleep while taking a screenshot

On MacOS, GrabViewSnapshot() call would block for the duration of
display/GPU sleep which leads to protocol client to wait and tests
to time out when power saving is engaged. This prevents power saving
while there are pending screenshot requests.

BUG= 599776 

Review-Url: https://codereview.chromium.org/2044783002
Cr-Commit-Position: refs/heads/master@{#398245}

[modify] https://crrev.com/2d6574bd49f71c153f24f0e78671fbcb481a2aaa/content/browser/renderer_host/render_widget_host_impl.cc
[modify] https://crrev.com/2d6574bd49f71c153f24f0e78671fbcb481a2aaa/content/browser/renderer_host/render_widget_host_impl.h

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/11d078ce662fd79da785df0565b3c30482e05a51

commit 11d078ce662fd79da785df0565b3c30482e05a51
Author: kbr <kbr@chromium.org>
Date: Wed Jun 08 02:35:51 2016

Fix chrome_telemetry_build's GN dependencies for Mac OS X.

(taking over from caseq's https://codereview.chromium.org/2037773005/ )

The previous bug workaround for  Issue 536192  should no longer be
necessary.

Expand OWNERS for chrome/BUILD.gn to include GN OWNERS.
Relax .gni ownership.

BUG= 599776 

Review-Url: https://codereview.chromium.org/2048563002
Cr-Commit-Position: refs/heads/master@{#398466}

[modify] https://crrev.com/11d078ce662fd79da785df0565b3c30482e05a51/chrome/BUILD.gn
[modify] https://crrev.com/11d078ce662fd79da785df0565b3c30482e05a51/chrome/OWNERS
[modify] https://crrev.com/11d078ce662fd79da785df0565b3c30482e05a51/tools/perf/chrome_telemetry_build/BUILD.gn

There was an underlying problem with the bot which will be eventually monitored in Issue 618123 but the above workaround should eliminate the flakiness. Thanks caseq@ for diagnosing this. Closing as fixed.