Flaky CHECK in blink::DevToolsSession::IOSession::DispatchProtocolCommand in webgl2_conformance_tests on Mac NVIDIA |
|||||
Issue descriptionDifferent tests fail: WebglConformance_conformance2_state_gl_object_get_calls - https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28NVIDIA%29/529 WebglConformance_conformance2_textures_canvas_tex_3d_r16f_red_half_float - https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28NVIDIA%29/520 WebglConformance_conformance2_textures_canvas_sub_rectangle_tex_2d_rgb5_a1_rgba_unsigned_short_5_5_5_1 - https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28NVIDIA%29/512 WebglConformance_conformance2_state_gl_object_get_calls - https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28NVIDIA%29/507 WebglConformance_conformance2_textures_canvas_tex_3d_r16f_red_half_float - https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28NVIDIA%29/488 WebglConformance_conformance2_query_occlusion_query - https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28NVIDIA%29/450 WebglConformance_conformance2_query_occlusion_query - https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28NVIDIA%29/449 WebglConformance_conformance2_state_gl_object_get_calls - https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28NVIDIA%29/444 [32123:19203:1204/084547.312374:FATAL:devtools_session.cc(72)] Check failed: false. 0 Chromium Framework 0x0000000118cf343f base::debug::StackTrace::StackTrace(unsigned long) + 31 1 Chromium Framework 0x0000000118bf170f logging::LogMessage::~LogMessage() + 223 2 Chromium Framework 0x000000011dbf77cc blink::DevToolsSession::IOSession::DispatchProtocolCommand(int, WTF::String const&, WTF::String const&) + 348 3 Chromium Framework 0x0000000116b499bd blink::mojom::blink::DevToolsSessionStubDispatch::Accept(blink::mojom::blink::DevToolsSession*, mojo::Message*) + 381 4 Chromium Framework 0x0000000118d67136 mojo::InterfaceEndpointClient::HandleValidatedMessage(mojo::Message*) + 982 5 Chromium Framework 0x0000000118d669e6 mojo::FilterChain::Accept(mojo::Message*) + 150 6 Chromium Framework 0x0000000118d68515 mojo::InterfaceEndpointClient::HandleIncomingMessage(mojo::Message*) + 117 7 Chromium Framework 0x0000000118d6ebab mojo::internal::MultiplexRouter::ProcessIncomingMessage(mojo::internal::MultiplexRouter::MessageWrapper*, mojo::internal::MultiplexRouter::ClientCallBehavior, base::SequencedTaskRunner*) + 1451 8 Chromium Framework 0x0000000118d6e019 mojo::internal::MultiplexRouter::Accept(mojo::Message*) + 377 9 Chromium Framework 0x0000000118d669e6 mojo::FilterChain::Accept(mojo::Message*) + 150 10 Chromium Framework 0x0000000118d61930 mojo::Connector::ReadSingleMessage(unsigned int*) + 576 11 Chromium Framework 0x0000000118d624c1 mojo::Connector::ReadAllAvailableMessages() + 97 12 Chromium Framework 0x0000000118d62379 mojo::Connector::OnHandleReadyInternal(unsigned int) + 137 13 Chromium Framework 0x00000001154795e7 mojo::SimpleWatcher::DiscardReadyState(base::RepeatingCallback<void (unsigned int)> const&, unsigned int, mojo::HandleSignalsState const&) + 103 14 Chromium Framework 0x0000000118d8a5ad mojo::SimpleWatcher::OnHandleReady(int, unsigned int, mojo::HandleSignalsState const&) + 365 15 Chromium Framework 0x0000000118d8a9e6 mojo::SimpleWatcher::Context::Notify(unsigned int, MojoHandleSignalsState, unsigned int) + 854 16 Chromium Framework 0x0000000118d895d3 mojo::SimpleWatcher::Context::CallNotify(MojoTrapEvent const*) + 35 17 Chromium Framework 0x00000001159786b7 mojo::core::WatcherDispatcher::InvokeWatchCallback(unsigned long, unsigned int, mojo::core::HandleSignalsState const&, unsigned int) + 119 18 Chromium Framework 0x0000000115977d12 mojo::core::Watch::InvokeCallback(unsigned int, mojo::core::HandleSignalsState const&, unsigned int) + 194 19 Chromium Framework 0x00000001159727b3 mojo::core::RequestContext::~RequestContext() + 1027 20 Chromium Framework 0x000000011596268e mojo::core::NodeChannel::OnChannelMessage(void const*, unsigned long, std::__1::vector<mojo::PlatformHandle, std::__1::allocator<mojo::PlatformHandle> >) + 302 21 Chromium Framework 0x000000011594eaa3 mojo::core::Channel::OnReadComplete(unsigned long, unsigned long*) + 835 22 Chromium Framework 0x000000011597ec90 mojo::core::(anonymous namespace)::ChannelPosix::OnFileCanReadWithoutBlocking(int) + 640 23 Chromium Framework 0x0000000118d1555c base::MessagePumpLibevent::OnLibeventNotification(int, short, void*) + 348 24 Chromium Framework 0x0000000118d16cbc event_base_loop + 1196 25 Chromium Framework 0x0000000118d15867 base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) + 455 26 Chromium Framework 0x0000000118c0b988 base::MessageLoopImpl::Run(bool) + 136 27 Chromium Framework 0x0000000118c46b39 base::RunLoop::Run() + 249 28 Chromium Framework 0x0000000118cb826e base::Thread::Run(base::RunLoop*) + 206 29 Chromium Framework 0x0000000118cb87eb base::Thread::ThreadMain() + 843 30 Chromium Framework 0x0000000118d015bf base::(anonymous namespace)::ThreadFunc(void*) + 95 31 libsystem_pthread.dylib 0x00007fff6ffba33d _pthread_body + 126 32 libsystem_pthread.dylib 0x00007fff6ffbd2a7 _pthread_start + 70 33 libsystem_pthread.dylib 0x00007fff6ffb9425 thread_start + 13 Not sure if there is a systematic issue here that affects random tests, or these specific tests are flaky and should be marked as such. Ken, PTAL Similar to issue 825338, not sure if related.
,
Dec 4
Here's the stack trace from within Telemetry's code which is provoking the renderer process crash:
Traceback (most recent call last):
_RunGpuTest at content/test/gpu/gpu_tests/gpu_integration_test.py:155
self.RunActualGpuTest(url, *args)
RunActualGpuTest at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:190
getattr(self, test_name)(test_path, *args[1:])
_RunConformanceTest at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:210
self._CheckTestCompletion()
_CheckTestCompletion at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:200
'webglTestHarness._finished', timeout=self._GetTestTimeout())
traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
return func(*args, **kwargs)
WaitForJavaScriptCondition at third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py:263
return self._tab.WaitForJavaScriptCondition(*args, **kwargs)
traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
return func(*args, **kwargs)
WaitForJavaScriptCondition at third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py:245
return self._inspector_backend.WaitForJavaScriptCondition(*args, **kwargs)
traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
return func(*args, **kwargs)
WaitForJavaScriptCondition at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:300
return py_utils.WaitFor(IsJavaScriptExpressionTrue, timeout)
WaitFor at third_party/catapult/common/py_utils/py_utils/__init__.py:138
res = condition()
IsJavaScriptExpressionTrue at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:297
return self._EvaluateJavaScript(condition, context_id, timeout)
traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
return func(*args, **kwargs)
Inner at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:38
return func(inspector_backend, *args, **kwargs)
_EvaluateJavaScript at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:552
self._runtime.Crash(context_id, timeout)
Crash at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_runtime.py:82
res = self._inspector_websocket.SyncRequest(request, timeout)
SyncRequest at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py:132
res = self._Receive(timeout)
_Receive at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py:190
self._HandleNotification(result)
_HandleNotification at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py:203
self._domain_handlers[domain_name](result)
traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
return func(*args, **kwargs)
_HandleInspectorDomainNotification at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:484
raise exception
DevtoolsTargetCrashException: Devtools target crashed
Let me try adding a piece of logic to try to crash the GPU process from a new tab in the browser when this happens.
,
Dec 5
Happens both on Mac FYI 10.14 Release (NVIDIA) and Mac FYI Experimental Retina Release (NVIDIA). https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28NVIDIA%29/2912 - WebglConformance_conformance2_glsl3_compound_assignment_type_combination BTW, main bot is 10.13.6 and experimental is also 10.13.6, so it makes sense to me to turn 10.14 bot into experimental bot and reduce the load on Macs from two 10.13.6 bots.
,
Dec 5
It turns out I'm really swamped this week - Yuly, is there any chance you could try suppressing these failures on Mojave on the appropriate hardware?
,
Dec 5
BTW, the logic I had in mind to add was: in webgl_conformance_integration_test.py, catch DevToolsTargetCrashException inside _CheckTestCompletion, and if it's raised, try making a new tab, navigating it to about:gpucrash, and close it. Any chance you could try writing that?
,
Dec 5
Re #5, that's beyond my expertise. Re #4, sure, I'll add the suppressions, if you think the problem is in these specific tests.
,
Dec 6
Since I'm going to add suppressions, I've decided to look for failures more thoroughly. So: WebglConformance_conformance2_glsl3_compound_assignment_type_combination https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28NVIDIA%29/2912 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28NVIDIA%29/2899 WebglConformance_conformance2_query_occlusion_query https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Retina%20Release%20%28NVIDIA%29/4150 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28NVIDIA%29/2865 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28NVIDIA%29/450 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28NVIDIA%29/449 WebglConformance_conformance2_state_gl_object_get_calls https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Retina%20Release%20%28NVIDIA%29/4157 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Retina%20Release%20%28NVIDIA%29/4148 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Retina%20Release%20%28NVIDIA%29/4065 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28NVIDIA%29/2890 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28NVIDIA%29/2760 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28NVIDIA%29/529 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28NVIDIA%29/507 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28NVIDIA%29/444 WebglConformance_conformance2_textures_canvas_sub_rectangle_tex_2d_r8_red_unsigned_byte https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28NVIDIA%29/2907 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28NVIDIA%29/2903 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28NVIDIA%29/2895 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28NVIDIA%29/2888 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28NVIDIA%29/2840 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28NVIDIA%29/2834 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28NVIDIA%29/2829 WebglConformance_conformance2_textures_canvas_sub_rectangle_tex_2d_rgb5_a1_rgba_unsigned_short_5_5_5_1 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28NVIDIA%29/2894 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28NVIDIA%29/512 WebglConformance_conformance2_textures_canvas_sub_rectangle_tex_2d_rgb16f_rgb_half_float https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28NVIDIA%29/2855 WebglConformance_conformance2_textures_canvas_sub_rectangle_tex_2d_rgb565_rgb_unsigned_short_5_6_5 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Retina%20Release%20%28NVIDIA%29/4165 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Retina%20Release%20%28NVIDIA%29/4158 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28NVIDIA%29/2850 WebglConformance_conformance2_textures_canvas_tex_3d_r16f_red_half_float https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Retina%20Release%20%28NVIDIA%29/4122 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28NVIDIA%29/534 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28NVIDIA%29/520 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28NVIDIA%29/488 Since most of the failures are both on 10.13.6 and 10.14, going to suppress all of the above on all mac nvidia.
,
Dec 7
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a30107e85221e2ce1afcfc58d7c9941c26b6ab5e commit a30107e85221e2ce1afcfc58d7c9941c26b6ab5e Author: Yuly Novikov <ynovikov@chromium.org> Date: Thu Dec 06 20:03:57 2018 Suppress Flaky WebGL and WebGL2 failures Android Nvidia: conformance/glsl/constructors/glsl-construct-mat2.html Win Intel OpenGL Passthrough: conformance2/rendering/out-of-bounds-index-buffers-after-copying.html Mac Nvidia: conformance2/glsl3/compound-assignment-type-combination.html conformance2/query/occlusion-query.html conformance2/state/gl-object-get-calls.html conformance2/textures/canvas_sub_rectangle/tex-2d-r8-red-unsigned_byte.html conformance2/textures/canvas_sub_rectangle/tex-2d-rgb5_a1-rgba-unsigned_short_5_5_5_1.html conformance2/textures/canvas_sub_rectangle/tex-2d-rgb16f-rgb-half_float.html' conformance2/textures/canvas_sub_rectangle/tex-2d-rgb565-rgb-unsigned_short_5_6_5.html conformance2/textures/canvas/tex-3d-r16f-red-half_float.html TBR=kbr@chromium.org Bug: 911772, 912579, 912583 Change-Id: I5ff7c6521d998502f5716ca014878e0aa07a30c8 Reviewed-on: https://chromium-review.googlesource.com/c/1366022 Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Yuly Novikov <ynovikov@chromium.org> Cr-Commit-Position: refs/heads/master@{#614462} [modify] https://crrev.com/a30107e85221e2ce1afcfc58d7c9941c26b6ab5e/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py [modify] https://crrev.com/a30107e85221e2ce1afcfc58d7c9941c26b6ab5e/content/test/gpu/gpu_tests/webgl_conformance_expectations.py
,
Dec 7
Thank you Yuly for helping with these suppressions. I'll still look into adding code which can get a stack trace out of a hung GPU process affecting these tests.
,
Dec 7
Yes, would be great if you could fix it. |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by kbr@chromium.org
, Dec 4Components: Internals>GPU>Testing Internals>GPU>Internals