Detected 3 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Some recent flakes were due to V8 GC bugs, now fixed.

Analysis of a few recent ones:

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/304539 ->  Issue 609252 

https://build.chromium.org/p/tryserver.chromium.android/builders/android_optional_gpu_tests_rel/builds/269 -> Telemetry's HTTP server seemed to crash. It looks like this was happening for a while on this bot but that it's fixed now.

https://build.chromium.org/p/tryserver.chromium.win/builders/win_optional_gpu_tests_rel/builds/4071 , https://build.chromium.org/p/tryserver.chromium.win/builders/win_optional_gpu_tests_rel/builds/4066 : possibly related to fix for  Issue 607283 , commented there

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/303047 : GPU process crash, unknown whether it's related to  Issue 609252 

Detected 5 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

The following Windows builds:
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/304053
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/303881
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/302630

all seem like instances of  Issue 609252 . They were failures in WebglConformance_conformance_extensions_oes_texture_half_float and all have this stack trace:

[3372:728:1003/102118:ERROR:gpu_watchdog_thread.cc(371)] The GPU process hung. Terminating after 15000 ms.

Backtrace:
	gpu::GpuWatchdogThread::DeliberatelyTerminateToRecoverFromHang [0x6A0BA8F9+688]
	??$MakeItSo@ABQ8WebMediaPlayerMS@content@@AEXXZABV?$WeakPtr@VWebMediaPlayerMS@content@@@base@@$$V@?$InvokeHelper@$00X@internal@base@@SAXABQ8WebMediaPlayerMS@content@@AEXXZABV?$WeakPtr@VWebMediaPlayerMS@content@@@2@@Z [0x69D6E603+38]
	base::internal::Invoker<base::internal::BindState<void (__thiscall content::WebMediaPlayerMS::*)(void),base::WeakPtr<content::WebMediaPlayerMS> >,void __cdecl(void)>::Run [0x69D73578+19]
	base::debug::TaskAnnotator::RunTask [0x688223FB+267]
	base::MessageLoop::RunTask [0x687E8B73+1187]
	base::MessageLoop::DeferOrRunPendingTask [0x687E783C+60]
	base::MessageLoop::DoDelayedWork [0x687E7B74+196]
	base::MessagePumpDefault::Run [0x68824C13+131]
	base::MessageLoop::RunHandler [0x687E86C0+96]
	base::RunLoop::Run [0x687F6999+41]
	base::Thread::Run [0x687F5CDD+173]
	base::Thread::ThreadMain [0x687F65AE+622]
	base::PlatformThread::Sleep [0x687C8902[3372:3176:1003/102131:ERROR:gles2_cmd_decoder.cc(4185)] [.Offscreen-For-WebGL-04D1A6C0]GL ERROR :GL_INVALID_OPERATION : glGetIntegerv: framebuffer incomplete
+290]
	BaseThreadInitThunk [0x76B6338A+18]
	RtlInitializeExceptionChain [0x77939902+99]
	RtlInitializeExceptionChain [0x779398D5+54]

I think the reference to WebMediaPlayerMS is spurious.

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/306962 , https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/306352 , https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/306022 , and https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/305989 were also  Issue 609252 .

https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/307210 was  Issue 638069 , just fixed.

Detected 3 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 5 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 7 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

GPU process hangs ( Issue 609252 ):
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/319609
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/319617
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/316501
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/316525
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/316963

Assertion failure after GPU process crash (probably because of hang);  Issue 609252 ,  Issue 648377 :
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/319689
[10039:1295:1020/171300:FATAL:DrawingBuffer.cpp(727)] Check failed: produceSyncToken.HasData(). 
WebglConformance_conformance_textures_webgl_canvas_tex_2d_rgba_rgba_unsigned_byte

Oilpan crash ( Issue 649485 ):
https://build.chromium.org/p/tryserver.chromium.win/builders/win_chromium_rel_ng/builds/316668

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/51e832338dfaeedf418778541a10003fd1feef8b

commit 51e832338dfaeedf418778541a10003fd1feef8b
Author: kbr <kbr@chromium.org>
Date: Tue Oct 25 02:34:54 2016

Try disabling GPU watchdog in WebGL conformance tests.

It's believed that the flaky GPU process hangs seen on the waterfall
are spurious, not real. Try disabling the GPU watchdog. If the hangs
are real, this will result in test timeouts rather than GPU process
crashes, context loss, and test failures.

BUG= 596622 ,  609252 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel

Review-Url: https://codereview.chromium.org/2446033002
Cr-Commit-Position: refs/heads/master@{#427247}

[modify] https://crrev.com/51e832338dfaeedf418778541a10003fd1feef8b/content/test/gpu/gpu_tests/webgl_conformance_integration_test.py

Detected 3 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/03ca36583160b14b922b4697cd63afd573abb5d5

commit 03ca36583160b14b922b4697cd63afd573abb5d5
Author: kbr <kbr@chromium.org>
Date: Mon Oct 31 00:04:26 2016

Symbolize all unsymbolized minidumps upon unexpected test failure.

This should provide reasonable diagnosis of the random GPU process
crahes seen running the WebGL conformance tests.

BUG= 596622 ,  619264 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=vmiura@chromium.org,zmo@chromium.org

Review-Url: https://codereview.chromium.org/2459213003
Cr-Commit-Position: refs/heads/master@{#428636}

[modify] https://crrev.com/03ca36583160b14b922b4697cd63afd573abb5d5/content/test/gpu/gpu_tests/gpu_integration_test.py

 Issue 534557  has been merged into this issue.

Detected 4 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

The last 4 flakes happened because the Windows GPU bots in the Swarming pool became overloaded. This pool is being expanded in Issue 631845.

Detected 3 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 7 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 4 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 10 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 8 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 11 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 5 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Detected 14 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

It looks like most or all of these new flakes are  Issue 666061 , being worked on.

Detected 5 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

At least 4 of the 5 recent failures (the fifth was affected by  Issue 667475 , so not sure) will be fixed with the fix for  Issue 666061 , which is in the CQ now.

Detected 4 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

One of those 4 was the original crash from  Issue 666061 .

One was the follow-on assertion failure fixed in https://codereview.chromium.org/2548153002 .

One was a failure of the Swarming shard on Android.

One was a GPU process crash which couldn't be symbolized because of  Issue 671049 .

Detected 3 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

https://build.chromium.org/p/tryserver.chromium.android/builders/android_optional_gpu_tests_rel/builds/1256 failed with "device '00ba5f938dbe26a3' not found" error. Not sure what happened there, is it related to master restart or not. Maybe lab issue? The device seem to have run some tests before disappearing, like WebglConformance_conformance_glsl_bugs_sequence_operator_evaluation_order.

All of the new (real) flakiness is a new assertion failure affecting WebglConformance_conformance_context_context_lost_restored . Filed  Issue 671942  about it.

Detected 4 new flakes for test/step "webgl_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM. This message was posted automatically by the chromium-try-flakes app.

Those four new flakes in #241 were  Issue 671942 , now fixed

I had hoped to close this as fixed and start a new bug to track future flakiness in this test suite, but more issues came up recently.  Issue 671791  is a real bug but hasn't been figured out yet.

At this point, the number of flakes per chromium-try-flakes are down to one or fewer per day:
https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyLwsSBUZsYWtlIiR3ZWJnbF9jb25mb3JtYW5jZV90ZXN0cyAod2l0aCBwYXRjaCkM

and the most recent failures of WebglConformance_conformance_context_context_release_with_workers are  Issue 674388 .

I'm declaring victory and closing this longstanding bug as fixed. Turning it around so that the still-outstanding bugs are blocked on this one, rather than blocking this on them.