WebGL guys, could you guys triage this CL?

There are a large number of flakes in the WebGL conformance tests, and they often get lumped together into the same bug, regardless of whether it's the same test failing or not.

Since the last TryFlakes-generated bug, two weeks ago, there have been sporadic failures on both Mac and Win builders, on ATI, NVIDIA, and Intel GPUs, on these tests:

1x - WebglConformance_conformance_ogles_GL_acos_acos_001_to_006 (Mac, NVIDIA)
1x - WebglConformance_conformance_ogles_GL_build_build_041_to_048 (Mac, Intel)
4x - WebglConformance_conformance_uniforms_no_over_optimization_on_uniform_array_12 (Mac, NVIDIA)
1x - WebglConformance_conformance2_textures_canvas_tex_2d_srgb8_rgb_unsigned_byte (Win, ATI)
1x - WebglConformance_deqp_functional_gles3_fbocolorbuffer_texcube_01 (Mac, ATI)
3x - WebglConformance_deqp_functional_gles3_sync (Win, NVIDIA)
1x - WebglConformance_deqp_functional_gles3_texturefiltering_2d_array_formats_07 (Win, ATI)

There's no real pattern to these failures (WebglConformance_deqp_functional_gles3_sync does seem to come up more often than anything else, but that's about it)

Do the WebGL folks have a handle on why these are collectively so flaky, and what can be done? It doesn't make sense to disable individual tests, when the next flake is usually on a completely different one, and disabling the whole suite seems drastic.

(Assigning to someone concrete for visiblity)

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

One of the Mac fyi bots appears to be failing almost every time with this:
https://build.chromium.org/p/chromium.gpu.fyi/builders/Mac%20GPU%20ASAN%20Release

Detected 5 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

WebglConformance_deqp_functional_gles3_sync is suppressed in  issue 676848 

iclelland@: the WebGL team as a whole strives to drive flakiness of these tests down to zero. Many recent flakes were caused by other components of the browser such as V8. Please see the chain of dependent bugs linked to this one.

I haven't had time over the last two days to triage the new reports; appreciate others' help including ynovikov@ and current pixel wrangler brianderson@. It looks like one test, WebglConformance_deqp_functional_gles3_sync, has been causing problems recently.

Brian, could you please look at the individual failures?

WebglConformance_deqp_functional_gles3_multisample failed once in https://build.chromium.org/p/tryserver.chromium.win/builders/win_optional_gpu_tests_rel/builds/7068 , for example. Win/AMD has been a configuration where in the past all WebGL tests were flaky -- not sure whether that's supposed to have been fixed by recent driver upgrades.

Note also that the Mac bots were just upgraded to macOS 10.12.2 in order to pick up several graphics driver bug fixes. It is possible that there will be some test fallout from that upgrade.

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Chris, as pixel wrangler could you please triage these most recent flakes? Thank you.

The Mac failures are
WebglConformance_conformance2_rendering_framebuffer_texture_level1

AssertionError: gl.checkFramebufferStatus(gl.FRAMEBUFFER) should be 36053. Was 36061.
FAIL gl.checkFramebufferStatus(gl.FRAMEBUFFER) should be 36053. Was 36061.

Which is to say that a framebuffer was reported as GL_FRAMEBUFFER_UNSUPPORTED when it was expected to be GL_FRAMEBUFFER_COMPLETE

Thanks Chris. That failure was due to a refactoring of a test, which required test expectations to be updated in Issue 678526.

The one Linux flake from yesterday:
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_optional_gpu_tests_rel/builds/6113

was a failure in WebglConformance_conformance2_textures_image_bitmap_from_image_bitmap_tex_2d_srgb8_alpha8_rgba_unsigned_byte . It looks like the ImageBitmap created from the ImageData was blank. This would probably be a race condition in this ImageBitmap constructor:
  ImageBitmap::ImageBitmap(
    ImageData* data,
    Optional<IntRect> cropRect,
    const ImageBitmapOptions& options)

I scanned through the code and didn't see anything obviously wrong. Maybe there's some incorrect synchronization in the GPU service side between the context which creates the (texture-backed) StaticBitmapImage, and the WebGL context, which consumes it and does a GPU-to-GPU copy to populate the WebGL texture. If it happens again I'll file a bug.

iclelland@, to further answer your question from #2 above, these tests act as end-to-end tests that exercise many areas of the browser's stack, including loading, the entire graphics stack from the low-level GPU code all the way up to Blink, V8, and other areas. We do aim to have fewer than 1 flake per day on the trybots, and aim for hundreds of consecutive green builds. We aim to not mark tests flaky, but do occasionally use that mechanism.

I'll take this bug and if the flakiness is back to an acceptable level next week will close it.

WebglConformance_deqp_functional_gles3_multisample flaked again in this try run:
https://build.chromium.org/p/tryserver.chromium.win/builders/win_optional_gpu_tests_rel/builds/7499

Shard:
https://chromium-swarm.appspot.com/task?id=34785b337fb89b10&refresh=10&show_raw=1

The test timed out. It looks like this is the first test that ran in this shard and that the browser didn't come up correctly. I'll try adding a flaky suppression on this configuration.

WebglConformance_deqp_functional_gles3_multisample (gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest) ... 
Traceback (most recent call last):
  _RunGpuTest at content\test\gpu\gpu_tests\gpu_integration_test.py:73
    self.RunActualGpuTest(url, *args)
  RunActualGpuTest at content\test\gpu\gpu_tests\webgl_conformance_integration_test.py:203
    getattr(self, test_name)(test_path, *args[1:])
  _RunConformanceTest at content\test\gpu\gpu_tests\webgl_conformance_integration_test.py:217
    self._CheckTestCompletion()
  _CheckTestCompletion at content\test\gpu\gpu_tests\webgl_conformance_integration_test.py:211
    'webglTestHarness._finished', timeout=300)
  traced_function at third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py:52
    return func(*args, **kwargs)
  WaitForJavaScriptCondition2 at third_party\catapult\telemetry\telemetry\internal\actions\action_runner.py:264
    return self.WaitForJavaScriptCondition(*args, **kwargs)
  traced_function at third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py:52
    return func(*args, **kwargs)
  WaitForJavaScriptCondition at third_party\catapult\telemetry\telemetry\internal\actions\action_runner.py:252
    return self._tab.WaitForJavaScriptCondition(*args, **kwargs)
  traced_function at third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py:52
    return func(*args, **kwargs)
  WaitForJavaScriptCondition at third_party\catapult\telemetry\telemetry\internal\browser\web_contents.py:191
    return self._inspector_backend.WaitForJavaScriptCondition(*args, **kwargs)
  traced_function at third_party\catapult\common\py_trace_event\py_trace_event\trace_event_impl\decorators.py:52
    return func(*args, **kwargs)
  WaitForJavaScriptCondition at third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py:281
    return py_utils.WaitFor(IsJavaScriptExpressionTrue, timeout)
  WaitFor at third_party\catapult\common\py_utils\py_utils\__init__.py:120
    res = condition()
  IsJavaScriptExpressionTrue at third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_backend.py:278
    return self._runtime.Evaluate(condition, context_id, timeout)
  Evaluate at third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_runtime.py:45
    res = self._inspector_websocket.SyncRequest(request, timeout)
  SyncRequest at third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_websocket.py:110
    res = self._Receive(timeout)
  _Receive at third_party\catapult\telemetry\telemetry\internal\backends\chrome_inspector\inspector_websocket.py:149
    data = self._socket.recv()
  recv at third_party\catapult\telemetry\third_party\websocket-client\websocket\_core.py:293
    opcode, data = self.recv_data()
  recv_data at third_party\catapult\telemetry\third_party\websocket-client\websocket\_core.py:310
    opcode, frame = self.recv_data_frame(control_frame)
  recv_data_frame at third_party\catapult\telemetry\third_party\websocket-client\websocket\_core.py:323
    frame = self.recv_frame()
  recv_frame at third_party\catapult\telemetry\third_party\websocket-client\websocket\_core.py:357
    return self.frame_buffer.recv_frame()
  recv_frame at third_party\catapult\telemetry\third_party\websocket-client\websocket\_abnf.py:336
    self.recv_header()
  recv_header at third_party\catapult\telemetry\third_party\websocket-client\websocket\_abnf.py:286
    header = self.recv_strict(2)
  recv_strict at third_party\catapult\telemetry\third_party\websocket-client\websocket\_abnf.py:371
    bytes_ = self.recv(min(16384, shortage))
  _recv at third_party\catapult\telemetry\third_party\websocket-client\websocket\_core.py:427
    return recv(self.sock, bufsize)
  recv at third_party\catapult\telemetry\third_party\websocket-client\websocket\_socket.py:83
    raise WebSocketTimeoutException(message)
WebSocketTimeoutException: timed out

Note: the earlier failure of WebglConformance_deqp_functional_gles3_multisample in:
https://build.chromium.org/p/tryserver.chromium.win/builders/win_optional_gpu_tests_rel/builds/7068

was a different issue -- it was a flaky failure of one of the individual tests:

Init testcase: multisample.fbo_max_samples.constancy_sample_coverage_inverted
Querying maximum number of samples for gl.RGBA8 with gl.getInternalformatParameter()
Using FBO of size (512, 512) with 8 samples
Clearing color to black
gl.SAMPLE_COVERAGE is enabled
Drawing several green quads, each fully overlapped by a red quad with the same inverted sample coverage values
Failure: Non-zero green color component detected - should have been completely overwritten by red quad
FAIL Failed

#17: This is interesting. The following tests are (VERY) flaky on Pixel:
multisample.fbo_max_samples.constancy_sample_coverage
multisample.fbo_max_samples.constancy_sample_coverage_inverted
multisample.fbo_max_samples.constancy_both
multisample.fbo_max_samples.constancy_both_inverted

But they seem to never fail on Firefox on Pixel, so it does seem likely that Chrome has some bug here.

Deleted: Screenshot_20170210-164010.png 168 KB

	Screenshot_20170210-164010.png 168 KB View Download

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/31a6a9f2ad1c4611de27af5c130077b293953c0e

commit 31a6a9f2ad1c4611de27af5c130077b293953c0e
Author: kbr <kbr@chromium.org>
Date: Wed Feb 22 05:49:35 2017

Mark deqp/functional/gles3/multisample.html flaky on Win/AMD R7 240.

BUG= 687374 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
TBR=zmo@chromium.org
NOTRY=true

Review-Url: https://codereview.chromium.org/2711463004
Cr-Commit-Position: refs/heads/master@{#451892}

[modify] https://crrev.com/31a6a9f2ad1c4611de27af5c130077b293953c0e/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py

Is this issue still Pri-1?

At this point some of the bots, like Mac Release (Intel), which run webgl2_conformance_tests have 200 green runs:
https://luci-milo.appspot.com/buildbot/chromium.gpu.fyi/Mac%20Release%20%28Intel%29/?limit=200

Unfortunately chromium-try-flakes doesn't seem to be working any more for this test suite.

I'm having difficulty getting reasonable results out of the flakiness dashboard. There are a lot of skipped tests in this suite (e.g. the builtinprecision tests) and they seem to dominate the tests that are reported.

I do see some failures there but they mainly seem to be on newly added tests and ones where there were recent changes, causing brief failures.

https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=webgl2_conformance_tests%20(with%20patch)&result=FAIL&builder=tryserver.chromium.linux%3Alinux_optional_gpu_tests_rel

As far as I can tell the major flakiness problems with this test suite have been addressed. If anyone finds otherwise please either reopen this bug or file a new one and, ideally, block it on this one.