Also: I triaged several of the failures of WebglConformance_deqp_functional_gles3_shaderloop_do_while and they don't appear to be correlated with any particular piece of hardware. Our code must be flaky somehow, but provoking the failure reliably seems difficult.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b57df21a12bb64a4e685641a7b11472efb07c1a9

commit b57df21a12bb64a4e685641a7b11472efb07c1a9
Author: kbr <kbr@chromium.org>
Date: Fri Aug 05 03:16:30 2016

Restart browser after expected failure.

Occasionally, an expected failure is causing a timeout, which causes
the test afterward to fail. Restart the browser in this case for
better robustness.

BUG= 619264 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
TBR=zmo@chromium.org

Review-Url: https://codereview.chromium.org/2214333003
Cr-Commit-Position: refs/heads/master@{#409979}

[modify] https://crrev.com/b57df21a12bb64a4e685641a7b11472efb07c1a9/content/test/gpu/gpu_tests/gpu_integration_test.py

Awesome, this has fixed the flake on the Mac FYI bot on the waterfall that has been green for 40 runs now.

That's a huge relief.

I've attempted unsuccessfully to provoke the crashes in WebGLObject::deleteObject that have been seen on the bots. This patch causes a GPU process crash during the execution of glCreateGpuMemoryBufferImageCHROMIUM, leading to a GL_OUT_OF_MEMORY error like that seen in https://bugs.chromium.org/p/chromium/issues/detail?id=619264#c40 :

https://codereview.chromium.org/2226193002

It was run with:

./content/test/gpu/run_gpu_integration_test.py webgl_conformance --show-stdout --browser=release -v --extra-browser-args="--enable-logging=stderr --js-flags=--expose-gc" --webgl-conformance-version=2.0.0 --read-abbreviated-json-results-from=content/test/data/gpu/webgl2_conformance_tests_output.json --test-filter=\(WebglConformance_deqp_functional_gles3_shaderloop_do_while\|WebglConformance_deqp_functional_gles3_texturefiltering_3d_combinations_25\) > output.txt 2>&1

Note that if src/content/test/data/gpu/webgl2_conformance_tests_output.json is updated, the shard distributions will change, and these tests might not run back-to-back any more.

The exact number of successful allocations allowed in GpuMemoryBufferMessageFilter::OnCreateGpuMemoryBuffer (src/content/gpu/gpu_child_thread.cc) in that patch is significant. It looks like if the crash happens too late, then WebglConformance_deqp_functional_gles3_texturefiltering_3d_combinations_25 goes into an infinite loop. However, under no circumstance was I able to provoke the renderer process crash seen on the bots.

I'm downgrading this to P2 at this point, because it looks like the flakiness is basically under control, and more analysis is needed for remaining flakiness on the waterfall.

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/2730 and https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/2714 were random GPU process crashes. Need to gather symbolized stack traces for these. vmiura@ is working on this in  Issue 609252 .

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/2710 was a random flaky failure of one of the sub-tests in WebglConformance_deqp_functional_gles3_shaderindexing_mat_00 :

[3862:1299:0816/134717:INFO:CONSOLE(11)] "Init testcase: indexing.matrix_subscript.mat3x2_dynamic_write_dynamic_read_vertex", source:  (11)
[3862:1299:0816/134717:INFO:CONSOLE(11)] "", source:  (11)
[3862:1299:0816/134717:INFO:CONSOLE(11)] "FAIL Fail", source:  (11)


I don't have an explanation for why this would fail intermittently.

Detected 8 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Here's a brief summary of the flakes, this is a v8 roll (https://codereview.chromium.org/2274803002):


build: https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/2897
failed test: WebglConformance_deqp_functional_gles3_shaderindexing_mat_00
reason:
AssertionError: indexing.matrix_subscript.mat3x2_dynamic_write_dynamic_read_vertex: Fail

--------------------------

build: https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/2895
failed test: WebglConformance_deqp_data_gles3_shaders_invalid_texture_functions
reason:
AssertionError: invalid_texture_functions.textureproj_sampler2darrayshadow_vec4_fragment: Start testcase: invalid_texture_functions.textureproj_sampler2darrayshadow_vec4_fragment

--------------------------

builds: https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/2883
        https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/2871
failed test: general test failure
reason:
2016-08-23 22:53:18.801 Chromium[42846:392444] An uncaught exception was raised
2016-08-23 22:53:18.802 Chromium[42846:392444] Error (1000) creating CGSWindow
2016-08-23 22:53:18.802 Chromium[42846:392444] (
0   CoreFoundation                      0x00007fff93be003c __exceptionPreprocess + 172
1   Chromium Framework                  0x000000010d36cc24 chrome_browser_application_mac::(anonymous namespace)::ExceptionPreprocessor(objc_object*) + 308
2   libobjc.A.dylib                     0x00007fff8a07576e objc_exception_throw + 43
3   CoreFoundation                      0x00007fff93bdfeed +[NSException raise:format:] + 205
4   AppKit                              0x00007fff917f0206 _NXCreateWindowWithStyleMask + 457
5   AppKit                              0x00007fff917effa2 _NSCreateWindow + 187
6   AppKit                              0x00007fff9164dd8d -[NSWindow _commonAwake] + 1434
7   AppKit                              0x00007fff915da70a -[NSWindow _commonInitFrame:styleMask:backing:defer:] + 864
8   AppKit                              0x00007fff915d9ca0 -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 1487
9   AppKit                              0x00007fff915d96cb -[NSWindow initWithContentRect:styleMask:backing:defer:] + 45
10  Chromium Framework                  0x0000000110626f78 StatusBubbleMac::Create() + 184
11  Chromium Framework                  0x0000000110626e39 StatusBubbleMac::StatusBubbleMac(NSWindow*, objc_object*) + 153
12  Chromium Framework                  0x0000000110581a97 -[BrowserWindowController initWithBrowser:takeOwnership:] + 2231
13  Chromium Framework                  0x000000011058c352 BrowserWindow::CreateBrowserWindow(Browser*) + 50
14  Chromium Framework                  0x00000001103a7d3c Browser::Browser(Browser::CreateParams const&) + 2748
...

--------------------------

build: https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/2876
failed test: WebglConformance_deqp_functional_gles3_shaderoperator_common_functions
reason:
TimeoutException: Timed out while waiting 300s for IsJavaScriptExpressionTrue.

--------------------------

build: https://build.chromium.org/p/tryserver.chromium.win/builders/win_optional_gpu_tests_rel/builds/3173
failed test: WebglConformance_deqp_functional_gles3_negativetextureapi
reason:
AssertionError: negativeTextureApi.compressedteximage2d_max_width_height: Expected gl.INVALID_VALUE, but got gl.NO_ERROR.

--------------------------

build: https://build.chromium.org/p/tryserver.chromium.win/builders/win_optional_gpu_tests_rel/builds/3168
failed test: WebglConformance_conformance_more_functions_drawElementsBadArgs
reason:
AssertionError: testDrawElementsVBOMultiByte

--------------------------

build: https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/2859
failed test: WebglConformance_conformance2_textures_video_tex_2d_r16f_red_half_float
reason:
AssertionError: Unable to fetch WebGL rendering context for Canvas
FAIL Unable to fetch WebGL rendering context for Canvas
Uncaught TypeError: Cannot read property 'RED' of null

Vlad: thanks for triaging the recent failures. It's necessary to dig a little deeper in the logs. For example, on this build:

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/2859

and this failed test:

WebglConformance_conformance2_textures_video_tex_2d_r16f_red_half_float

digging a little earlier in the log indicates that the previous test crashed the GPU process, and apparently the test still passed. It's not clear from the logs whether that was a GPU watchdog timeout or not. We didn't get a symbolized stack trace.

Here's the log excerpt demonstrating this:

----------
WebglConformance_deqp_functional_gles3_texturespecification_texstorage2d_format_size (gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest) ... Received signal 11 SEGV_MAPERR 011b171c4000
 [0x000105558066]
 [0x7fff87fe7f1a]
 [0x00010d9f7218]
 [0x7fff8a4dd612]
 [0x7fff8a661799]
 [0x7fff8a661460]
 [0x7fff8a661b33]
 [0x7fff8a65f3dd]
[end of stack trace]
ok
WebglConformance_conformance2_textures_video_tex_2d_r16f_red_half_float (gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest) ... [17721:1299:0823/130053:ERROR:command_buffer_proxy_impl.cc(232)] Failed to send GpuChannelMsg_CreateCommandBuffer.
[17721:1299:0823/130053:ERROR:context_provider_command_buffer.cc(162)] GpuChannelHost failed to create command buffer.
[17717:1299:0823/130053:INFO:CONSOLE(11)] "Unable to fetch WebGL rendering context for Canvas", source:  (11)
[17717:1299:0823/130053:INFO:CONSOLE(11)] "FAIL Unable to fetch WebGL rendering context for Canvas", source:  (11)
[17717:1299:0823/130053:INFO:CONSOLE(11)] "Uncaught TypeError: Cannot read property 'RED' of null", source:  (11)
----------

Could you dig a little deeper to see whether the other failures have the same root cause?

We need to do the following:

1) Instrument webgl_conformance_integration_test.py with code that will use the new Telemetry APIs for discovering and symbolizing unsymbolized minidumps, to see if we can get stack traces for these GPU process crashes.

2) Perhaps -- and very unfortunately -- mark all of the WebGL 2.0 conformance tests as flaky, so they get retried, at least, if this happens.

WebglConformance_conformance_misc_delayed_drawing failure on Mac Retina AMD is due to GPU process crash from the previous test

WebglConformance_deqp_functional_gles3_transformfeedback_array_element_separate_lines (gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest) ... Received signal 11 SEGV_MAPERR 0000ffffffff
 [0x00010d9da736]
 [0x7fff8a794f1a]
 [0xffff80059d9aff70]
 [0x7fff84bfd612]
 [0x7fff8c726799]
 [0x7fff8c726460]
 [0x7fff8c726b33]
 [0x7fff8c7243dd]
[end of stack trace]
WARNING:root:WebglConformance_deqp_functional_gles3_transformfeedback_array_element_separate_lines was expected to fail, but passed. 

Another Mac Retina AMD failure: shaderindexing_mat_00

AssertionError: indexing.matrix_subscript.mat3x2_dynamic_write_dynamic_read_vertex: Fail

Detected 5 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Suppressing failure of WebglConformance_conformance2_misc_getextension_while_pbo_bound_stability on Linux in  Issue 644572 .

The other recent flakes look like renderer process crashes for which no symbols were generated -- this is very unfortunate.

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_optional_gpu_tests_rel/builds/2682 : random failure of WebglConformance_deqp_functional_gles3_texturespecification_random_teximage2d_2d . Not sure what is going on there.

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/3468/ : possibly  Issue 609252 .

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/3452/ :  Issue 645280  (really, Issue 644669, for which a workaround was just landed).

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_optional_gpu_tests_rel/builds/2709 and https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_optional_gpu_tests_rel/builds/2688 => Issue 618447 , suppressing in https://codereview.chromium.org/2350503002/

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/3486 -> random timeout of WebglConformance_conformance2_textures_video_tex_2d_r8_red_unsigned_byte , not sure what happened. Let's watch for this again.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d42e1b0d09fc6ad1a7d2c5ef8e57887716049163

commit d42e1b0d09fc6ad1a7d2c5ef8e57887716049163
Author: kbr <kbr@chromium.org>
Date: Sat Sep 17 00:14:42 2016

Suppress flaky texturespecification/random_teximage2d_2d.html on Linux NVIDIA.

BUG=618447,  619264 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=cwallez@chromium.org, zmo@chromium.org
NOTRY=true

Review-Url: https://codereview.chromium.org/2350503002
Cr-Commit-Position: refs/heads/master@{#419339}

[modify] https://crrev.com/d42e1b0d09fc6ad1a7d2c5ef8e57887716049163/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py

Both recent new flakes:
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/3515
and
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/3511

are instances of  Issue 609252  -- though I filed Issue 648318 about the former, since the test harness apparently went into an infinite loop after the context was lost.

Detected 4 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

The flakes above looked mainly like Swarming capacity issues with the NVIDIA Mac Retina bots. If they persist we may need to take these tests off the mac_optional_gpu_tests_rel tryserver.

Analysis of a few recent failures:

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/3763 , https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/3757 : GPU process crashes, unsure whether related to  Issue 609252 

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/3758 : capacity issue with NVIDIA Retina MBPs

Detected 4 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/3763 , https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/3757 -> GPU process crash;  issue 609252 ?

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/3777 -> V8 assertion failure:
WebglConformance_deqp_functional_gles3_fboinvalidate_format_01 (gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest) ... 

#
# Fatal error in ../../v8/src/heap/spaces.h, line 1639
# Check failed: size_ >= bytes (12226328 vs. 172630272).
#

==== C stack trace ===============================

    0   Chromium Framework                  0x00000001062e8ec3 v8::base::debug::StackTrace::StackTrace() + 19
    1   Chromium Framework                  0x00000001062e4e0d V8_Fatal + 221
    2   Chromium Framework                  0x0000000105e73fd4 v8::internal::PagedSpace::EmptyAllocationInfo() + 548
    3   Chromium Framework                  0x0000000105e7b869 v8::internal::FreeList::Allocate(int) + 89
    4   Chromium Framework                  0x0000000105daca93 v8::internal::PagedSpace::AllocateRawUnaligned(int, v8::internal::PagedSpace::UpdateSkipList) + 67
    5   Chromium Framework                  0x0000000105dac7eb v8::internal::Heap::AllocateRaw(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) + 475
    6   Chromium Framework                  0x0000000105de709f v8::internal::Heap::AllocateFillerObject(int, bool, v8::internal::AllocationSpace) + 31
    7   Chromium Framework                  0x0000000105d95c7e v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) + 46
    8   Chromium Framework                  0x00000001060ae1f4 v8::internal::__RT_impl_Runtime_AllocateInTargetSpace(v8::internal::Arguments, v8::internal::Isolate*) + 132
    9   Chromium Framework                  0x00000001060adeb3 v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) + 259
    10  ???                                 0x00000ce516709f46 0x0 + 14178063523654

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/3788 , https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/3786 -> capacity issues on Mac NVIDIA Retina bots

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/3822 ->  Issue 609252 

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

The three failures mentioned above included 2 V8 crashes (apparently now fixed) and an apparent GPU process hang.

The three most recent flakes:

https://build.chromium.org/p/tryserver.chromium.win/builders/win_optional_gpu_tests_rel/builds/4393
https://build.chromium.org/p/tryserver.chromium.win/builders/win_optional_gpu_tests_rel/builds/4385
https://build.chromium.org/p/tryserver.chromium.win/builders/win_optional_gpu_tests_rel/builds/4355

were all instances of  Issue 609252 .

Ken: to make this less painful, I think we can add s.t that convert the merged simplified result format to full json test results formart

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 5 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

https://build.chromium.org/p/tryserver.chromium.win/builders/win_optional_gpu_tests_rel/builds/4684 , https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/4333 :  Issue 659165 

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/4335 : some random infra issue where the whole machine failed. Not sure what happened, let's see if it resurfaces.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/03ca36583160b14b922b4697cd63afd573abb5d5

commit 03ca36583160b14b922b4697cd63afd573abb5d5
Author: kbr <kbr@chromium.org>
Date: Mon Oct 31 00:04:26 2016

Symbolize all unsymbolized minidumps upon unexpected test failure.

This should provide reasonable diagnosis of the random GPU process
crahes seen running the WebGL conformance tests.

BUG= 596622 ,  619264 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel
TBR=vmiura@chromium.org,zmo@chromium.org

Review-Url: https://codereview.chromium.org/2459213003
Cr-Commit-Position: refs/heads/master@{#428636}

[modify] https://crrev.com/03ca36583160b14b922b4697cd63afd573abb5d5/content/test/gpu/gpu_tests/gpu_integration_test.py

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 10 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Those flakes were  Issue 662069 .

In a try job for:
https://codereview.chromium.org/2481403002
specifically
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/4678
WebglConformance_deqp_functional_gles3_transformfeedback_array_element_separate_lines
failed because the GPU process crashed randomly.

A background thread running QuartzCore code (CoreAnimation?) crashed:
Thread 4 (crashed)
 0  QuartzCore + 0xc211
 1  QuartzCore + 0x25e3c
 2  libsystem_pthread.dylib + 0x53dc
 3  libsystem_pthread.dylib + 0x4f63
 4  libsystem_pthread.dylib + 0x3582
 5  libsystem_pthread.dylib + 0x1341
 6  libdispatch.dylib + 0x5aa5

Erik, Chris, have you seen an issue like this before?

Deleted: fullstack.txt 71.7 KB

fullstack.txt 71.7 KB View Download

Hmm, no ideas offhand what could be causing that.

Grabbed the relevant libraries from the machine to get a more accurate stacktrace. Not very interesting:
"""
_free_magazine:
_free_allocator_thread:
__pthread_exit
__pthread_exit
__pthread_wqthread
_start_wqthread
"""

Grabbed the system log. Only thing that stands out. Wonder if there's a hardware overheating issue? Thermal state 2 is "serious". Thermal state 3 is "critical". https://developer.apple.com/reference/foundation/processinfo.thermalstate/
"""
Nov  7 16:37:46 build488-m4 com.apple.CDScheduler[237]: Thermal pressure state: 0 Memory pressure state: 0
...
Nov  7 16:39:15 build488-m4 com.apple.CDScheduler[237]: Thermal pressure state: 1 Memory pressure state: 0
...
Nov  7 16:40:39 build488-m4 com.apple.CDScheduler[237]: Thermal pressure state: 2 Memory pressure state: 0
[Radio silence for twenty seconds]
Nov  7 16:41:01 build488-m4 kernel[0]: Chromium Helper[8012] triggered ...
"""

Looking at the swarming logs:
https://chromium-swarm.appspot.com/user/task/325a2e2f64239810

"""
WebglConformance_deqp_functional_gles3_texturespecification_teximage3d_depth_pbo (gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest) ... [8004:1295:1107/164059:INFO:CONSOLE(73)] "GL_INVALID_OPERATION : glTexImage3D: invalid unpack params combination", source: http://127.0.0.1:54123/third_party/webgl/src/sdk/tests/deqp/framework/opengl/simplereference/sglrGLContext.js (73)
[8004:1295:1107/164100:INFO:CONSOLE(73)] "GL_INVALID_OPERATION : glTexImage3D: invalid unpack params combination", source: http://127.0.0.1:54123/third_party/webgl/src/sdk/tests/deqp/framework/opengl/simplereference/sglrGLContext.js (73)
[8004:1295:1107/164100:INFO:CONSOLE(73)] "GL_INVALID_OPERATION : glTexImage3D: invalid unpack params combination", source: http://127.0.0.1:54123/third_party/webgl/src/sdk/tests/deqp/framework/opengl/simplereference/sglrGLContext.js (73)
[8004:1295:1107/164100:INFO:CONSOLE(73)] "GL_INVALID_OPERATION : glTexImage3D: invalid unpack params combination", source: http://127.0.0.1:54123/third_party/webgl/src/sdk/tests/deqp/framework/opengl/simplereference/sglrGLContext.js (73)
[8004:1295:1107/164100:INFO:CONSOLE(73)] "GL_INVALID_OPERATION : glTexImage3D: invalid unpack params combination", source: http://127.0.0.1:54123/third_party/webgl/src/sdk/tests/deqp/framework/opengl/simplereference/sglrGLContext.js (73)
ok
WebglConformance_deqp_functional_gles3_transformfeedback_array_element_separate_lines (gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest) ... Received signal 11 SEGV_MAPERR 0000ffffffff
 [0x00010420ec06]
 [0x7fff8a2e452a]
 [0x7fa8e0e5cfe0]
 [0x7fff9c963e3c]
 [0x7fff897733dc]
 [0x7fff89772f63]
 [0x7fff89771582]
 [0x7fff8976f341]
[end of stack trace]
"""

Hm, looks like there's a test success right before our crash, but a bunch of GL errors. This always makes me suspicious. Any possibility we're leaving the GL driver in a bad state from the previous tests?

Deleted: build488-m4_system_log.zip 263 KB

build488-m4_system_log.zip 263 KB Download

(Going to reduce the CC: list in the next update -- folks, feel free to add yourselves back on)

Erik: thanks for grabbing the system log. It's very interesting that you identified a potential overheating problem because that's been found to be an issue on a couple of the other GPU bots recently.

Peter, John, Bryce: do you think there's a possibility of build488-m4 running hot? This is in the golo, right? Could someone spot-check the rack? Thanks.

Erik: regarding OpenGL errors: each test like WebglConformance_deqp_functional_gles3_transformfeedback_array_element_separate_lines creates a new OpenGL context unrelated to previous ones, so I don't think state from one test is affecting the next.

re: temperature, the cpu temp of build488-m4 (according to an unproven utility) reads about 105.4°F idle (no tasks). This is about the same as my laptop at idle, for comparison. 

During task https://chromium-swarm.appspot.com/user/task/325ab195498bc411 it went up to 146.1°F @ 19:02:16 PST

Things are pretty idle on the bot now, but I imagine with back to back tasks, things could get a bit warmer. I can do another sample over time tomorrow. Keep in mind there's not a whole lot we can do about ambient temperatures in this datacenter. 

Thanks

webgl_conformance_tests time stamped run (similar behaviour)

https://chromium-swarm.appspot.com/user/task/325b1e8f4511bc11

Mon Nov  7 20:56:51 PST 2016
104.7°F
Mon Nov  7 20:57:21 PST 2016
115.7°F
Mon Nov  7 20:57:51 PST 2016
123.1°F
Mon Nov  7 20:58:21 PST 2016
122.5°F
Mon Nov  7 20:58:51 PST 2016
123.8°F
Mon Nov  7 20:59:21 PST 2016
124.5°F
Mon Nov  7 20:59:51 PST 2016
130.1°F
Mon Nov  7 21:00:21 PST 2016
137.1°F
Mon Nov  7 21:00:51 PST 2016
142.0°F
Mon Nov  7 21:01:21 PST 2016
145.2°F
Mon Nov  7 21:01:51 PST 2016
146.3°F
Mon Nov  7 21:02:21 PST 2016
135.9°F
Mon Nov  7 21:02:51 PST 2016
133.9°F
Mon Nov  7 21:03:21 PST 2016
129.0°F
Mon Nov  7 21:03:51 PST 2016
131.2°F
Mon Nov  7 21:04:21 PST 2016
137.5°F
Mon Nov  7 21:04:51 PST 2016
136.2°F
Mon Nov  7 21:05:21 PST 2016
134.2°F
Mon Nov  7 21:05:51 PST 2016
123.4°F
Mon Nov  7 21:06:21 PST 2016
118.2°F
Mon Nov  7 21:06:51 PST 2016
115.0°F
Mon Nov  7 21:07:21 PST 2016
113.7°F
Mon Nov  7 21:07:51 PST 2016
113.0°F
Mon Nov  7 21:08:21 PST 2016
111.9°F
Mon Nov  7 21:08:51 PST 2016
111.0°F
Mon Nov  7 21:09:21 PST 2016
110.5°F
Mon Nov  7 21:09:51 PST 2016
109.9°F
Mon Nov  7 21:10:21 PST 2016
109.0°F
Mon Nov  7 21:10:51 PST 2016
108.7°F

build488-m4 is a macbookpro.  I think they are racked 2 to a shelf, one in front with the other behind, with the laptops facing towards the front of the rack.  These racks are on a raised floor, cooling tile in front of the rack.

Would not surprise me at all if this is one is one of the ones in the back of a shelf. 
We would nee someone on the floor to check.

We should compare temperatures to similar positioned macbookpros.

Thanks John and Peter for the amazing detailed measurements and the thought. I filed a separate ticket Issue 663577 about checking that one particular machine.

I investigated another of the flakes and filed Issue 663601 about it to separate out the crashes inside QuartzCore / Core Animation from this bug.

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 7 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 4 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 6 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 4 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

Detected 6 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

There were a lot of new failures over the past couple of days caused by  Issue 671791  and  Issue 672367 . Let's let things stabilize again and re-evaluate the stability.

Saw this here:
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/5649
Task page:
https://chromium-swarm.appspot.com/user/task/32fe0179734b6110


WebglConformance_conformance_textures_image_tex_2d_rgba_rgba_unsigned_short_5_5_5_1 (gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest) ... 
Traceback (most recent call last):
_RunGpuTest at content/test/gpu/gpu_tests/gpu_integration_test.py:78
self.RunActualGpuTest(url, *args)
RunActualGpuTest at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:196
getattr(self, test_name)(test_path, *args[1:])
_RunConformanceTest at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:210
self._CheckTestCompletion()
_CheckTestCompletion at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:204
'webglTestHarness._finished', timeout_in_seconds=300)
traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
return func(*args, **kwargs)
WaitForJavaScriptCondition at third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py:246
self._tab.WaitForJavaScriptExpression(condition, timeout_in_seconds)
traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
return func(*args, **kwargs)
WaitForJavaScriptExpression at third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py:143
e.message + '\n' + debug_message)
TimeoutException: Timed out while waiting 300s for IsJavaScriptExpressionTrue.
Console output:

Locals:
IsJavaScriptExpressionTrue : <function IsJavaScriptExpressionTrue at 0x10f97de60>
debug_message : 'Console output:\n'
e : TimeoutException('Timed out while waiting 300s for IsJavaScriptExpressionTrue.',)
expr : 'webglTestHarness._finished'
timeout : 300

Timeouts in conformance/textures/image can be an indication that HTMLImageElements instantiated via the "new Image()" constructor are being GC'd before their onload handler is called.

Detected 3 new flakes for test/step "webgl2_conformance_tests (with patch)". To see the actual flakes, please visit https://chromium-try-flakes.appspot.com/all_flake_occurrences?key=ahVzfmNocm9taXVtLXRyeS1mbGFrZXNyMAsSBUZsYWtlIiV3ZWJnbDJfY29uZm9ybWFuY2VfdGVzdHMgKHdpdGggcGF0Y2gpDA. This message was posted automatically by the chromium-try-flakes app.

The flakiness of this test suite is looking really good at this point. Any flakes are occurring at a rate of less than 1 per day. This seems satisfactory enough to call this fixed, and to open a new bug if the flakiness spikes. Thanks everyone for your help getting to this point. I'm turning around the blocking/blocked-on dependencies for the remaining bugs.