New issue
Advanced search Search tips

Issue 902406 link

Starred by 1 user
Status: Fixed
Owner:
rjkroege@chromium.org
Closed: Nov 21
Cc:
weiliangc@chromium.org
rjkroege@chromium.org
thomasanderson@chromium.org
nedngu...@google.com
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug-Regression



Sign in to add a comment

webgl2_conformance_tests failing on chromium.gpu.fyi/Linux FYI Release (NVIDIA)

Project Member Reported by sheriff-...@appspot.gserviceaccount.com, Nov 6 
Filed by sheriff-o-matic@appspot.gserviceaccount.com on behalf of weiliangc@chromium.org

webgl2_conformance_tests failing on chromium.gpu.fyi/Linux FYI Release (NVIDIA)

Builders failed on: 
- Linux FYI Release (NVIDIA): 
  https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20FYI%20Release%20%28NVIDIA%29

Example of failing build:
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20FYI%20Release%20%28NVIDIA%29/6174

Seems to be test time out:
[14/139] gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest.WebglConformance_conformance2_textures_canvas_tex_3d_rgb16f_rgb_float failed unexpectedly 321.6208s:
  No path to crashpad_database_util found
  No minidump found via crashpad_database_util
  No path to crashpad_database_util found
  No minidump found via crashpad_database_util
  Can't get standard output with --show-stdout
  
  Traceback (most recent call last):
    _RunGpuTest at content/test/gpu/gpu_tests/gpu_integration_test.py:155
      self.RunActualGpuTest(url, *args)
    RunActualGpuTest at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:189
      getattr(self, test_name)(test_path, *args[1:])
    _RunConformanceTest at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:203
      self._CheckTestCompletion()
    _CheckTestCompletion at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:197
      'webglTestHarness._finished', timeout=self._GetTestTimeout())
    traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
      return func(*args, **kwargs)
    WaitForJavaScriptCondition at third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py:261
      return self._tab.WaitForJavaScriptCondition(*args, **kwargs)
    traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
      return func(*args, **kwargs)
    WaitForJavaScriptCondition at third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py:245
      return self._inspector_backend.WaitForJavaScriptCondition(*args, **kwargs)
    traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
      return func(*args, **kwargs)
    WaitForJavaScriptCondition at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:307
      self.GetCurrentConsoleOutputBuffer())
    traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
      return func(*args, **kwargs)
    Inner at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:41
      inspector_backend._ConvertExceptionFromInspectorWebsocket(e)
    traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
      return func(*args, **kwargs)
    Inner at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:38
      return func(inspector_backend, *args, **kwargs)
    GetCurrentConsoleOutputBuffer at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:196
      return self._console.GetCurrentConsoleOutputBuffer(timeout)
    GetCurrentConsoleOutputBuffer at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_console.py:38
      self._inspector_websocket.DispatchNotifications(timeout)
    DispatchNotifications at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py:156
      self._Receive(timeout)
    _Receive at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py:181
      raise WebSocketException(err)
  TimeoutException: Timeout after 300s while waiting for JavaScript:webglTestHarness._finished
  
  Exception thrown when trying to capture console output: TimeoutException('',)
  
  Locals:
    err     : WebSocketTimeoutException('timed out',)
    timeout : 10
  
  No path to crashpad_database_util found
  No minidump found via crashpad_database_util
  Restarting browser due to unexpected test failure
  Closing browser (pid=10224) ...
  Failed to gracefully shutdown.
  Proceed to kill the browser.
  Browser is closed.

Comment 1 by dtapu...@chromium.org, Nov 15

Components: Blink>WebGL
Status: Untriaged (was: Available)

Comment 2 by kbr@chromium.org, Nov 16

Blockedon: 905519 905511
Cc: rjkroege@chromium.org thomasanderson@chromium.org nedngu...@google.com
Labels: -Pri-2 Pri-1 Type-Bug-Regression
Owner: rjkroege@chromium.org
This is distressing; there's no obvious failure mode in the logs.

It's hard to see whether this is a widespread issue because of other flakiness like  Issue 905519 .

rjkroege@: as current pixel wrangler could you please take a look at this? This bot's become unacceptably flaky recently - it's failing nearly all runs right now. Some of these flakes are caused by  Issue 905519 , some seem caused by  Issue 905511  (now fixed); other causes as yet unknown.

Comment 4 by vmi...@chromium.org, Nov 16

Status: Assigned (was: Untriaged)

Comment 5 by rjkroege@chromium.org, Nov 20

Blockedon: 906889 906846
I looked through the failures and discovered two new failures that I'd missed underneath several other overlapping failures. I've now filed bugs and suppressions for 906889 and 906889.

Summary;

* 6247 WebglConformance_conformance2_extensions_webgl_multiview. GPU crash. 905519 (fix landed)
* 6248 WebglConformance_conformance2_extensions_webgl_multiview, GPU crash, 905519 (fix landed)
* 6259 WebglConformance_conformance2_extensions_webgl_multiview, GPU crash, 905519 (fix landed)
* 6267 WebglConformance_conformance2_extensions_webgl_multiview, GPU crash, 905519 (fix landed)
* 6278 WebglConformance_conformance2_extensions_webgl_multiview, GPU crash, 905519 (fix landed)
* 6279 WebglConformance_conformance2_extensions_webgl_multiview, GPU crash, 905519 (fix landed)
* 6283 passthrough WebglConformance_deqp_functional_gles3_multisample, blink GC crash, 904546 (fix landed)
* 6284 WebglConformance_conformance2_extensions_webgl_multiview, GPU crash, 905519 (fix landed)
* 6285 WebglConformance_conformance2_extensions_webgl_multiview, GPU crash, 905519 (fix landed)
* 6288 WebglConformance_conformance2_extensions_webgl_multiview, GPU crash, 905519 (fix landed)
* 6292 WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level, https://chromium-review.googlesource.com/c/1325197, fix landed
* 6293 WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level
* 6294 passthrough WebglConformance_conformance2_rendering_multisampling_fragment_evaluation GPU crash, 905511, (fix landed)
* 6294 WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level GPU crash, 905511, (fix landed)
* 6296 passthrough WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level GPU crash, 905511, (fix landed)
* 6296  WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level 
* 6299 passthrough WebglConformance_conformance2_rendering_multisampling_fragment_evaluation GPU crash, 905511, (fix landed)
* 6299   WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level, GPU crash, 905511, (fix landed)
* 6300 passthrough WebglConformance_conformance2_rendering_multisampling_fragment_evaluation GPU crash, 905511, (fix landed)
* 6300 WebglConformance_conformance2_extensions_webgl_multiview, GPU crash, 905519 (fix landed)
* 6301 passthrough WebglConformance_conformance2_rendering_multisampling_fragment_evaluation GPU crash, 905511, (fix landed)
* 6301 WebglConformance_conformance2_extensions_webgl_multiview, GPU crash, 905519 (fix landed)
* 6302 WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level, GPU crash, 905519 (fix landed)
* 6303 WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level, GPU crash, 905519 (fix landed)
* 6304 passthrough WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level GPU crash, 905511, (fix landed)
* 6304 WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level, GPU crash, 905511, (fix landed)
* 6305 passthrough WebglConformance_conformance2_rendering_multisampling_fragment_evaluation GPU crash, 905511, (fix landed)
* 6306 passthrough WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level GPU crash, 905511, (fix landed)
* 6306 WebglConformance_conformance2_extensions_webgl_multiview, GPU crash, 905519 (fix landed)
* 6307 passthrough WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level GPU crash, 905511, (fix landed)
* 6309 passthrough WebglConformance_deqp_functional_gles3_multisample, blink GC crash, 904546 (fix landed)
* 6309 passthrough WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level GPU crash without stack trace. 905511, (fix landed) seems likely.
* 6309  WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level GPU crash, 905511, (fix landed)
* 6311  WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level GPU crash, 905511, (fix landed)
* 6313  WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level GPU crash, 905511, (fix landed)
* 6314 passthrough WebglConformance_conformance2_rendering_multisampling_fragment_evaluation GPU crash, 905511, (fix landed)
* 6314  WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level GPU crash, 905511, (fix landed)
* 6315 WebglConformance_conformance2_extensions_webgl_multiview, GPU crash, 905519 (fix landed)
* 6318  passthrough WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level GPU crash, 905511, (fix landed)
* 6320  WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level GPU crash, 905511, (fix landed)
* 6323  passthrough WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level GPU crash, 905511, (fix landed)
* 6324  WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level GPU crash, 905511, (fix landed)
* 6326  WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level GPU crash, 905511, (fix landed)
* 6327  passthrough WebglConformance_conformance2_rendering_multisampling_fragment_evaluation GPU crash, 905511, (fix landed)
* 6327 WebglConformance_conformance2_extensions_webgl_multiview, GPU crash, 905519 (fix landed)
* 6329  passthrough WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level GPU crash, 905511, (fix landed)
* 6329  WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level, no crash dump,  GPU crash, probably 905511, (fix landed)
* 6330 passthrough WebglConformance_conformance2_rendering_multisampling_fragment_evaluation, GPU crash, 905511 (fix landed)
* 6331 passthrough WebglConformance_conformance2_rendering_multisampling_fragment_evaluation, GPU crash, 905519 (fix landed)
* 6332 WebglConformance_conformance2_extensions_webgl_multiview, GPU crash, 905519 (fix landed)
* 6333 passthrough WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level, GPU crash, 905511 (fix landed)
* 6333 WebglConformance_conformance2_extensions_webgl_multiview, GPU crash, 905519 (fix landed)
* 6333 WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level, GPU crash, 905511 (fix landed)
* 6334 WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level, no crash dump,  GPU crash, probably 905511, (fix landed)
* 6335 passthrough WebglConformance_conformance2_rendering_multisampling_fragment_evaluation, GPU crash, 905511 (fix landed)
* 6335 WebglConformance_conformance2_extensions_webgl_multiview, GPU crash, 905519 (fix landed)
* 6336 WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level, GPU crash, 905511 (fix landed)
* 6337 WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level, GPU crash, 905511 (fix landed)
* 6338 passthrough WebglConformance_conformance2_rendering_multisampling_fragment_evaluation, GPU crash, 905511 (fix landed)
* 6340  passthrough WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level no crash dump,  GPU crash likely, probably 905511
* 6340 WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level, GPU crash, 905511 (fix landed)
* 6341 passthrough WebglConformance_conformance2_rendering_multisampling_fragment_evaluation, GPU crash, 905511 (fix landed)
* 6342  WebglConformance_conformance2_rendering_multisampling_fragment_evaluation, GPU crash, 905511 (fix landed)
* 6343 WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level, GPU crash, 905511 (fix landed)
* 6344 WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level, GPU crash, 905511 (fix landed)
* 6345 WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level, GPU crash, 905511 (fix landed)
* 6346 passthrough WebglConformance_conformance2_rendering_multisampling_fragment_evaluation, GPU crash, 905511 (fix landed)
* 6349 WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level, GPU crash, 905511 (fix landed)
* 6350 passthrough WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level, GPU crash, 905511 (fix landed)
* 6351 passthrough WebglConformance_conformance2_rendering_multisampling_fragment_evaluation, GPU crash, 905511 (fix landed)
* 6352 passthrough WebglConformance_conformance2_rendering_multisampling_fragment_evaluation, GPU crash, 905511 (fix landed)
* 6353 passthrough WebglConformance_conformance_extensions_webgl_compressed_texture_s3tc: swizzle error.  https://crbug.com/906882 
* 6354 passthrough WebglConformance_conformance2_rendering_multisampling_fragment_evaluation, GPU crash, 905511 (fix landed)
* 6356 Infra issue (tracked separately)
* 6335 WebglConformance_conformance2_extensions_webgl_multiview, GPU crash, 905519 (fix landed)
* 6359 passthrough WebglConformance_conformance2_textures_canvas_tex_3d_rgba16f_rgba_float, albeit no crash dump, likely 905511 (fix landed)
* 6359 WebglConformance_conformance2_extensions_webgl_multiview, GPU crash, 905519 (fix landed)
* 6359 WebglConformance_conformance2_rendering_draw_with_integer_texture_base_level, possibly GPU crash, 905511? Not obvious.
* 6364 passthrough WebglConformance_conformance2_rendering_canvas_resizing_with_pbo_bound suppression in 906846 - driver bug exposed by WebGL roll https://chromium-review.googlesource.com/c/1335270?
* 6365 WebglConformance_conformance2_rendering_canvas_resizing_with_pbo_bound, GPU crash, 905519 (fix landed)
* 6366 WebglConformance_conformance2_rendering_blitframebuffer_test, GPU crash, 905519 (fix landed). But there is a different issue here. I suspect new WebGL tests again. 
* 6366  WebglConformance_conformance2_context_methods_2, GPU crash, 905519 (fix landed)
* 6368  WebglConformance_conformance2_context_methods_2, GPU crash, 905519 (fix landed)
* 6371 WebglConformance_conformance2_context_methods_2, GPU crash, 905519 (fix landed)
* 6375 WebglConformance_conformance2_rendering_canvas_resizing_with_pbo_bound, 906846
* 6377 WebglConformance_conformance2_context_methods_2, GPU crash, 905519 (fix landed)
* 6379 WebglConformance_conformance2_rendering_canvas_resizing_with_pbo_bound, 906846
* 6380 WebglConformance_conformance2_rendering_canvas_resizing_with_pbo_bound, 906846
* 6384 WebglConformance_conformance2_rendering_canvas_resizing_with_pbo_bound, 906846
* 6390 WebglConformance_conformance2_rendering_canvas_resizing_with_pbo_bound, 906846
* 6391 WebglConformance_conformance2_rendering_blitframebuffer_test, first fail not obscured by 905519
* 6400 WebglConformance_conformance2_rendering_canvas_resizing_with_pbo_bound, 906846
* 6403 WebglConformance_conformance2_rendering_canvas_resizing_with_pbo_bound, 906846
* 6405 WebglConformance_conformance2_rendering_canvas_resizing_with_pbo_bound, 906846
* 6407 WebglConformance_conformance2_rendering_blitframebuffer_test, 906889
* 6409 WebglConformance_conformance2_rendering_blitframebuffer_test, 906889
* 6414 WebglConformance_conformance2_rendering_canvas_resizing_with_pbo_bound, 906846
* 6417 WebglConformance_conformance2_rendering_canvas_resizing_with_pbo_bound, 906846
* 6420 WebglConformance_conformance2_rendering_blitframebuffer_test, 906889
* 6424 WebglConformance_conformance2_rendering_blitframebuffer_test, 906889
* 6428 WebglConformance_conformance2_rendering_canvas_resizing_with_pbo_bound, 906846
* 6429 WebglConformance_conformance2_rendering_canvas_resizing_with_pbo_bound, 906846
* 6430 WebglConformance_conformance2_rendering_canvas_resizing_with_pbo_bound, 906846
* 6424 WebglConformance_conformance2_rendering_blitframebuffer_test, 906889
* 6438 WebglConformance_conformance2_rendering_canvas_resizing_with_pbo_bound, 906846

Comment 6 by rjkroege@chromium.org, Nov 21

Blockedon: -905519 -906889 -906846 -905511
Status: Fixed (was: Assigned)
As of Wed Nov 21 10:26:21 EST 2018, Linux FYI Nvidia Rel has been green for 18 builds. 
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20FYI%20Release%20(NVIDIA) 6464-6481 passing.

I'm going to declare this bug fixed as in: the builder is operating as intended and detecting new failures.

Comment 7 by kbr@chromium.org, Nov 27 

Thank you Robert for getting to the bottom of this!
►

Sign in to add a comment