@kbr: Could you take a look at this?

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/220473 is an instance of  Issue 606112 .

https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/222534 was probably a transient issue while that machine had its drivers upgraded in Issue 605264. It should not occur again.

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/221584 and https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/223064 are a new apparent race condition, filed as  Issue 608946  .

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/11920935044f1f08dbed0dfe24656d7704d74553

commit 11920935044f1f08dbed0dfe24656d7704d74553
Author: Scott Little <sclittle@chromium.org>
Date: Wed May 04 00:16:07 2016

Disable flaky GpuCrash.GPUProcessCrashesExactlyOnce test on mac and linux.

TBR=kbr@chromium.org

BUG= 608923 
CQ_INCLUDE_TRYBOTS=tryserver.chromium.linux:linux_optional_gpu_tests_rel;tryserver.chromium.mac:mac_optional_gpu_tests_rel;tryserver.chromium.win:win_optional_gpu_tests_rel

Review URL: https://codereview.chromium.org/1947673003 .

Cr-Commit-Position: refs/heads/master@{#391405}

[modify] https://crrev.com/11920935044f1f08dbed0dfe24656d7704d74553/content/test/gpu/gpu_tests/context_lost_expectations.py

Detected 12 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Another suppression incoming in https://codereview.chromium.org/1947303002/ .

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0795777fbf9349b39b0c66704e5ce7633add15b8

commit 0795777fbf9349b39b0c66704e5ce7633add15b8
Author: kbr <kbr@chromium.org>
Date: Thu May 05 00:24:09 2016

Skip GpuCrash.GPUProcessCrashesExactlyOnce on all platforms.

The assertion failure in  Issue 608946  is firing frequently on the
commit queue, and the flaky retry mechanism isn't working for this
failure mode.

BUG= 608923 ,  608946 
CQ_INCLUDE_TRYBOTS=tryserver.chromium.linux:linux_optional_gpu_tests_rel;tryserver.chromium.mac:mac_optional_gpu_tests_rel;tryserver.chromium.win:win_optional_gpu_tests_rel
TBR=danakj@chromium.org, sclittle@chromium.org
NOTRY=true

Review-Url: https://codereview.chromium.org/1947303002
Cr-Commit-Position: refs/heads/master@{#391712}

[modify] https://crrev.com/0795777fbf9349b39b0c66704e5ce7633add15b8/content/test/gpu/gpu_tests/context_lost_expectations.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a76f4435305e2b1a7fa0c8a5e0c5ae9cdde9755c

commit a76f4435305e2b1a7fa0c8a5e0c5ae9cdde9755c
Author: danakj <danakj@chromium.org>
Date: Fri May 06 22:57:36 2016

Revert of Skip GpuCrash.GPUProcessCrashesExactlyOnce on all platforms. (patchset #1 id:1 of https://codereview.chromium.org/1947303002/ )

Reason for revert:
Temporary fix here: https://codereview.chromium.org/1949753007/

More work to come.

Original issue's description:
> Skip GpuCrash.GPUProcessCrashesExactlyOnce on all platforms.
>
> The assertion failure in  Issue 608946  is firing frequently on the
> commit queue, and the flaky retry mechanism isn't working for this
> failure mode.
>
> BUG= 608923 ,  608946 
> CQ_INCLUDE_TRYBOTS=tryserver.chromium.linux:linux_optional_gpu_tests_rel;tryserver.chromium.mac:mac_optional_gpu_tests_rel;tryserver.chromium.win:win_optional_gpu_tests_rel
> TBR=danakj@chromium.org, sclittle@chromium.org
> NOTRY=true
>
> Committed: https://crrev.com/0795777fbf9349b39b0c66704e5ce7633add15b8
> Cr-Commit-Position: refs/heads/master@{#391712}

TBR=sclittle@chromium.org,kbr@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 608923 ,  608946 

Review-Url: https://codereview.chromium.org/1956703007
Cr-Commit-Position: refs/heads/master@{#392193}

[modify] https://crrev.com/a76f4435305e2b1a7fa0c8a5e0c5ae9cdde9755c/content/test/gpu/gpu_tests/context_lost_expectations.py

Detected 3 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

The new flakes above are instances of  Issue 606112 .

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/889fa48ee6cbe56579a77382232eaef9293816c2

commit 889fa48ee6cbe56579a77382232eaef9293816c2
Author: dyen <dyen@chromium.org>
Date: Fri May 13 06:14:26 2016

Tentatively enable flaky GpuCrash.GPUProcessCrashesExactlyOnce again.

R=kbr@chromium.org
BUG= 608923 ,  606112 
CQ_INCLUDE_TRYBOTS=tryserver.chromium.linux:linux_optional_gpu_tests_rel;tryserver.chromium.mac:mac_optional_gpu_tests_rel;tryserver.chromium.win:win_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/1971403002
Cr-Commit-Position: refs/heads/master@{#393454}

[modify] https://crrev.com/889fa48ee6cbe56579a77382232eaef9293816c2/content/test/gpu/gpu_tests/context_lost_expectations.py

Detected 3 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Assigning to dyen to triage the recent failures.

Already assigned, no need for Sheriff label.

Detected 3 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

The three recent flakes are of GpuCrash.GPUProcessCrashesExactlyOnce:

https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/224347
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/223414
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/223446

They happen on Win/AMD and Win/NVIDIA.

One new failure mode seems to be in gpu::gles2::FeatureInfo::InitializeFeatures (though the stack could be somewhat bogus):

[4532:3724:0517/163538:ERROR:command_buffer_proxy_impl.cc(236)] Failed to send GpuChannelMsg_CreateCommandBuffer.
[4532:3724:0517/163538:ERROR:context_provider_command_buffer.cc(161)] GpuChannelHost failed to create command buffer.
[5116:5060:0517/163538:INFO:CONSOLE(0)] "WebGL: CONTEXT_LOST_WEBGL: loseContext: context lost", source: http://127.0.0.1:59054/gpu_process_crash.html (0)
Backtrace:
	(No symbol) [0x3036392E]
	(No symbol) [0x5BE434E1]
	gpu::gles2::FeatureInfo::InitializeFeatures [0x5CF645E4+4037]
	RtlInterlockedFlushSList [0x77B03417+3202]
	RtlInterlockedFlushSList [0x77B03022+2189]
	RtlInterlockedFlushSList [0x77B02FFE+2153]
	RtlAllocateHeap [0x77AFE0F2+172]
	HeapFree [0x76A614BD+20]
	free [0x5C5C292C+20]
	std::_Deallocate [0x5C5BC512+72]
	base::Time::NowFromSystemTime [0x5F2211FF+303]

...
[3240:2084:0517/163545:ERROR:command_buffer_proxy_impl.cc(236)] Failed to send GpuChannelMsg_CreateCommandBuffer.
[3240:2084:0517/163545:ERROR:context_provider_command_buffer.cc(161)] GpuChannelHost failed to create command buffer.
[5116:5060:0517/163545:INFO:CONSOLE(22)] "Failed to fetch WebGL context", source: http://127.0.0.1:59054/gpu_process_crash.html (22)
...
[2808:4668:0517/163545:ERROR:context_provider_command_buffer.cc(190)] Failed to initialize GLES2Implementation.
[4424:4852:0517/163545:ERROR:gles2_cmd_decoder.cc(15078)] [.Offscreen-MainThread-08367660]GL ERROR :GL_INVALID_OPERATION : glCreateAndConsumeTextureCHROMIUM: invalid mailbox name
[4424:4852:0517/163545:ERROR:gles2_cmd_decoder.cc(8222)] [.Offscreen-MainThread-08367660]RENDER WARNING: texture bound to texture unit 0 is not renderable. It maybe non-power-of-2 and have incompatible texture filtering.
Waiting for page to finish.
Waiting for page to finish.
Traceback (most recent call last):
  File "c:\users\chrome~1\appdata\local\temp\runuctgor\third_party\catapult\telemetry\telemetry\internal\story_runner.py", line 84, in _RunStoryAndProcessErrorIfNeeded
    state.RunStory(results)
  File "c:\users\chrome~1\appdata\local\temp\runuctgor\content\test\gpu\gpu_tests\gpu_test_base.py", line 111, in RunStory
    RunStoryWithRetries(GpuSharedPageState, self, results)
  File "c:\users\chrome~1\appdata\local\temp\runuctgor\content\test\gpu\gpu_tests\gpu_test_base.py", line 72, in RunStoryWithRetries
    super(cls, shared_page_state).RunStory(results)
  File "c:\users\chrome~1\appdata\local\temp\runuctgor\third_party\catapult\telemetry\telemetry\page\shared_page_state.py", line 332, in RunStory
    self._current_page, self._current_tab, results)
  File "c:\users\chrome~1\appdata\local\temp\runuctgor\content\test\gpu\gpu_tests\context_lost.py", line 118, in ValidateAndMeasurePage
    'Test failed (didn\'t render content properly?)')
Failure: Test failed (didn't render content properly?)

[  FAILED  ] GpuCrash.GPUProcessCrashesExactlyOnce (14232 ms)


Not sure what's going on here.

It's important that this test not be disabled completely because that will lose coverage. Perhaps it should be marked as flaky again on Windows.

Already assigned, no need for Sheriff label.

Detected 3 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Detected 3 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Recent flakes =>  Issue 613229 

Detected 9 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

These flakes are under the umbrella of  Issue 613229 .

Detected 5 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app. Since flakiness is ongoing, the issue was moved back into Sheriff Bug Queue (unless already there).

Detected 3 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

All of those flakes are due to lack of capacity on the Swarming Windows slaves, covered in Issue 614526.

Detected 5 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 27 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

The massive number of failures above seem to have been caused by the Swarming pool being overloaded. This should be alleviated soon by new machines coming online in Issue 614526.

Detected 3 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

The flakes in #38 were  Issue 617847 .

Detected 8 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Again,  Issue 617847 .

Detected 3 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Filed  Issue 619196  about these new flakes.

Detected 9 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 22 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Those flakes were all  Issue 619196 .

Detected 4 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Those flakes were capacity issues being addressed in Issue 614526.

Detected 3 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

The recent flakes of this test indicate that the browser is failing to start, which is a serious problem. Example failures of this kind:

https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/252901
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/252910
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/252768

It looks like the GPU process might be crashing based on the hardware_accelerated_feature_tests failures in https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/252768 .

enne, as pixel wrangler this week could you please look in the logs and changelogs from today to see if there are any suspect changes which might be taking down the browser, or evidence elsewhere on the waterfalls (maybe the Perf waterfall) of similar hangs? Thanks.

I'm no longer seeing these failures
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng

Should we close?

Detected 6 new flakes for test/step "context_lost_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyKgsSBUZsYWtlIh9jb250ZXh0X2xvc3RfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Since it looks like these flakes aren't happening any more, closing as WontFix (not reproducible). The flurry of flakes on August 11 were Issue 637081.