First draw after resizing WebGL canvas fails on NVIDIA GPUs on macOS 10.13 |
||||||||||||||||||||
Issue descriptionSee status in https://build.chromium.org/p/chromium.gpu.fyi/builders/Mac%20Experimental%20Retina%20Release%20%28NVIDIA%29?numbuilds=200 This has been happening on-and-off since https://build.chromium.org/p/chromium.gpu.fyi/builders/Mac%20Experimental%20Retina%20Release%20%28NVIDIA%29/builds/2510 r508506 through r508573 Culprit CL is not immediately obvious. This does not appear on other platforms The output is gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest.WebglConformance_conformance_canvas_drawingbuffer_static_canvas_test failed unexpectedly 1.7004s: Traceback (most recent call last): _RunGpuTest at content/test/gpu/gpu_tests/gpu_integration_test.py:125 self.RunActualGpuTest(url, *args) RunActualGpuTest at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:203 getattr(self, test_name)(test_path, *args[1:]) _RunConformanceTest at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:217 self._CheckTestCompletion() _CheckTestCompletion at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:213 self.fail(self._WebGLTestMessages(self.tab)) fail at /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py:412 raise self.failureException(msg) AssertionError: Line 15 should be red for at least 10 red pixels starting 20 pixels in at (20, 15) expected: 255,0,0,255 was 0,0,0,255 FAIL Line 15 should be red for at least 10 red pixels starting 20 pixels in at (20, 15) expected: 255,0,0,255 was 0,0,0,255 Locals: msg : u'Line 15 should be red for at least 10 red pixels starting 20 pixels in\nat (20, 15) expected: 255,0,0,255 was 0,0,0,255\nFAIL Line 15 should be red for at least 10 red pixels starting 20 pixels in\nat (20, 15) expected: 255,0,0,255 was 0,0,0,255\n'
,
Oct 16 2017
Will do!
,
Oct 16 2017
,
Oct 16 2017
I've added highsierra to test_expectations in https://chromium-review.googlesource.com/720164, BTW.
,
Oct 16 2017
Ah great, thanks Yuly. Note: possible reproduction steps: 1) Install Chrome Canary on macOS 2) Check out the root Chromium sources ( https://chromium.googlesource.com/chromium/src/ ) 3) Run: ./src/content/test/gpu/run_gpu_integration_test.py webgl_conformance --browser=canary --test-filter=WebglConformance_conformance_canvas_drawingbuffer_static_canvas_test Not sure how many times it would be necessary to repeat the steps above to potentially reproduce the failure.
,
Oct 17 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0fc6199fdea23f3137a79dbe97ca8a581a15bdad commit 0fc6199fdea23f3137a79dbe97ca8a581a15bdad Author: Christopher Cameron <ccameron@chromium.org> Date: Tue Oct 17 07:45:11 2017 Disable high sierra WebGL drawingbuffer conformance test Bug: 775202 Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Change-Id: I9565ee77b276fe14ecbdfaf17ec18d348a80cb85 Reviewed-on: https://chromium-review.googlesource.com/722034 Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: ccameron chromium <ccameron@chromium.org> Cr-Commit-Position: refs/heads/master@{#509316} [modify] https://crrev.com/0fc6199fdea23f3137a79dbe97ca8a581a15bdad/content/test/gpu/gpu_tests/webgl_conformance_expectations.py
,
Oct 18 2017
,
Oct 24 2017
Still seeing flakes from this. Here's a couple of builds that failed recently. https://build.chromium.org/p/chromium.gpu.fyi/builders/Mac%20Experimental%20Retina%20Release%20%28NVIDIA%29/builds/2610 https://build.chromium.org/p/chromium.gpu.fyi/builders/Mac%20Experimental%20Retina%20Release%20%28NVIDIA%29/builds/2614
,
Oct 25 2017
Should we disable it on https://build.chromium.org/p/chromium.gpu.fyi/builders/Mac%20Experimental%20Retina%20Release%20%28NVIDIA%29 as well? Its been flaking on it a lot.
,
Oct 27 2017
I can see the flakiness on an NVIDIA MacBook Pro running 10.13 by navigating to https://www.khronos.org/registry/webgl/sdk/tests/conformance/canvas/drawingbuffer-static-canvas-test.html on Chrome Canary. Unfortunately: passing --disable-features=WebGLImageChromium on the command line does not fix or work around the flakiness. Reloading the page multiple times seems to get it to start to render correctly. Khushal: the test expectation Chris added is supposed to be taking effect on the machine you point out. I'll test on the machine here and see why it seems to not be.
,
Oct 27 2017
The reason's that there's a separate expectations file for the webgl2_conformance_tests runs, and that's where this test is still failing.
,
Oct 27 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/7deb129693a5f99481b8616bd18a25d539d4828a commit 7deb129693a5f99481b8616bd18a25d539d4828a Author: Kenneth Russell <kbr@chromium.org> Date: Fri Oct 27 17:57:01 2017 Suppress drawingbuffer-static-canvas-test in WebGL 2.0 runs. TBR=ccameron@chromium.org BUG= 775202 Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Change-Id: Id004ed752567428e0100052d6590fa7970ae322f Reviewed-on: https://chromium-review.googlesource.com/741923 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#512231} [modify] https://crrev.com/7deb129693a5f99481b8616bd18a25d539d4828a/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py
,
Oct 27 2017
Calling this ExternalDependency until Apple fixes their 10.13 drivers.
,
Oct 27 2017
This test's failing in Safari too. There must be something broken with large texture allocations on this GPU. I proactively filed Radar 35224718 about it.
,
Oct 27 2017
,
Nov 1 2017
Note: this test can be easily run online at: https://www.khronos.org/registry/webgl/sdk/tests/conformance/canvas/drawingbuffer-static-canvas-test.html On 10.13 the canvas shows up black pretty frequently. Reloading the page eventually causes the red triangle to show up.
,
Nov 13 2017
The failure seems better understood now and seems to be a failure to render the first time after the WebGL canvas is resized, only on NVIDIA GPUs on macOS 10.13: not on other GPU types, and not on 10.12. It's pretty strange that the same failure is seen in both Chrome and Safari because the browser implementations differ so much, and also that it doesn't happen in Firefox on the same hardware. The three.js community has run into this issue: https://github.com/mrdoob/three.js/issues/12359 In particular with this test case: https://jsfiddle.net/f2Lommf5/109/ Gregg Tavares has contributed another small test case: https://github.com/KhronosGroup/WebGL/pull/2543 Continuing to try to isolate the failure.
,
Nov 13 2017
Issue 784057 has been merged into this issue.
,
Nov 18 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f9d1bf632952f7c909121b63b3b61183de7a8d53 commit f9d1bf632952f7c909121b63b3b61183de7a8d53 Author: Kenneth Russell <kbr@chromium.org> Date: Sat Nov 18 02:00:55 2017 Roll WebGL 12192b9..e4919fa https://chromium.googlesource.com/external/khronosgroup/webgl.git/+log/12192b9..e4919fa BUG= 775202 , 778770 , 784817 TEST=bots TBR=zmo@chromium.org, kainino@chromium.org NOTRY=true CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.win:win_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.android:android_optional_gpu_tests_rel Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Change-Id: I296437377a0d5084f2ede5f50d6b6519436bc62f Reviewed-on: https://chromium-review.googlesource.com/773625 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#517678} [modify] https://crrev.com/f9d1bf632952f7c909121b63b3b61183de7a8d53/DEPS [modify] https://crrev.com/f9d1bf632952f7c909121b63b3b61183de7a8d53/content/test/gpu/gpu_tests/webgl_conformance_expectations.py [modify] https://crrev.com/f9d1bf632952f7c909121b63b3b61183de7a8d53/content/test/gpu/gpu_tests/webgl_conformance_revision.txt
,
Nov 21 2017
,
Nov 21 2017
The problem appears to be a bug in the graphics driver related to the reallocation of depth-stencil renderbuffers at a different size. The framebuffer to which these are attached does not seem to correctly pick up the reallocated renderbuffer. Creating a new renderbuffer object behind the scenes and attaching it to all framebuffers pointing to the other renderbuffer -- applying the same fix for Issue 696126 -- works around the bug. Kudos to kainino@ for doing great infrastructure work in that bug. Unfortunately, I wasn't successful in reducing this to a standalone Objective-C++ test case. Attempt is attached, but it doesn't reproduce the bug. https://www.khronos.org/registry/webgl/sdk/tests/conformance/canvas/render-after-resize-test.html , however, does -- in both Chrome and Safari. Perhaps the bug is related to the use of Remote Core Animation in the same process, since Firefox is unaffected. (Firefox, however, always creates a new renderbuffer object when resizing its WebGL back buffer.) https://chromium-review.googlesource.com/780211 is up for review working around the bug.
,
Nov 21 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3b7911b65b0d70164ed9b88ec22a9a455fa649b7 commit 3b7911b65b0d70164ed9b88ec22a9a455fa649b7 Author: Kenneth Russell <kbr@chromium.org> Date: Tue Nov 21 20:04:16 2017 Add workaround for reallocating depth-stencil renderbuffers. When needed, during reallocation of a depth-stencil renderbuffer, create a new object and point all existing framebuffer objects to it. Apply this workaround on NVIDIA GPUs on macOS 10.13 and later. This works around an apparent bug in the graphics driver introduced in this release. This is a slight expansion of the earlier multisample_renderbuffer_resize_emulation workaround. Remove associated failure expectations. Also update the bug ID in the conformance expectations for a workaround applying to the same platform. BUG= 775202 , 786717 , 784817 Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Change-Id: I051009b17df1cb5dabe1b00fbdcc35008042eee9 Reviewed-on: https://chromium-review.googlesource.com/780211 Reviewed-by: Kai Ninomiya <kainino@chromium.org> Commit-Queue: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#518348} [modify] https://crrev.com/3b7911b65b0d70164ed9b88ec22a9a455fa649b7/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py [modify] https://crrev.com/3b7911b65b0d70164ed9b88ec22a9a455fa649b7/content/test/gpu/gpu_tests/webgl_conformance_expectations.py [modify] https://crrev.com/3b7911b65b0d70164ed9b88ec22a9a455fa649b7/gpu/command_buffer/service/gles2_cmd_decoder.cc [modify] https://crrev.com/3b7911b65b0d70164ed9b88ec22a9a455fa649b7/gpu/command_buffer/service/renderbuffer_manager.cc [modify] https://crrev.com/3b7911b65b0d70164ed9b88ec22a9a455fa649b7/gpu/command_buffer/service/renderbuffer_manager.h [modify] https://crrev.com/3b7911b65b0d70164ed9b88ec22a9a455fa649b7/gpu/config/gpu_driver_bug_list.json [modify] https://crrev.com/3b7911b65b0d70164ed9b88ec22a9a455fa649b7/gpu/config/gpu_driver_bug_workaround_type.h
,
Nov 21 2017
Given that this bug was P1, I think we should consider merging this fix back to M63. It's small and self-contained, well understood, and will ease pain for some 10.13 users on older NVIDIA MacBook Pros. I've received confirmation from NVIDIA that they simultaneously tracked down this bug in their driver, and that the diagnosis is correct.
,
Nov 21 2017
This bug requires manual review: We are only 13 days from stable. Please contact the milestone owner if you have questions. Owners: cmasso@(Android), cmasso@(iOS), gkihumba@(ChromeOS), govind@(Desktop) For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Nov 22 2017
Could you pls let us know which CL you're requesting merge for? Is the change well baked/verified in canary, having enough automation coverage and fully safe to merge to M63?
,
Nov 22 2017
Ping!
,
Nov 22 2017
Sorry, I'm requesting merge for: https://chromium-review.googlesource.com/780211 It's in Canary and I've just verified manually that the fix is working well. We've added automated regression tests for the bug. It's safe to merge to M63.
,
Nov 22 2017
,
Nov 27 2017
,
Nov 28 2017
Pls merge your change to M63 branch 3239 before 12:30 PM, Tuesday (11/28/17) so we can take it in for this week last Beta release. Thank you.
,
Nov 28 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/23a5326c8ae2d90654faeda8cbd7225b08ed3412 commit 23a5326c8ae2d90654faeda8cbd7225b08ed3412 Author: Kenneth Russell <kbr@chromium.org> Date: Tue Nov 28 19:03:46 2017 Add workaround for reallocating depth-stencil renderbuffers. When needed, during reallocation of a depth-stencil renderbuffer, create a new object and point all existing framebuffer objects to it. Apply this workaround on NVIDIA GPUs on macOS 10.13 and later. This works around an apparent bug in the graphics driver introduced in this release. This is a slight expansion of the earlier multisample_renderbuffer_resize_emulation workaround. Remove associated failure expectations. Also update the bug ID in the conformance expectations for a workaround applying to the same platform. BUG= 775202 , 786717 , 784817 TBR=kbr@chromium.org (cherry picked from commit 3b7911b65b0d70164ed9b88ec22a9a455fa649b7) Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Change-Id: I051009b17df1cb5dabe1b00fbdcc35008042eee9 Reviewed-on: https://chromium-review.googlesource.com/780211 Reviewed-by: Kai Ninomiya <kainino@chromium.org> Commit-Queue: Kenneth Russell <kbr@chromium.org> Cr-Original-Commit-Position: refs/heads/master@{#518348} Reviewed-on: https://chromium-review.googlesource.com/794024 Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/branch-heads/3239@{#584} Cr-Branched-From: adb61db19020ed8ecee5e91b1a0ea4c924ae2988-refs/heads/master@{#508578} [modify] https://crrev.com/23a5326c8ae2d90654faeda8cbd7225b08ed3412/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py [modify] https://crrev.com/23a5326c8ae2d90654faeda8cbd7225b08ed3412/content/test/gpu/gpu_tests/webgl_conformance_expectations.py [modify] https://crrev.com/23a5326c8ae2d90654faeda8cbd7225b08ed3412/gpu/command_buffer/service/gles2_cmd_decoder.cc [modify] https://crrev.com/23a5326c8ae2d90654faeda8cbd7225b08ed3412/gpu/command_buffer/service/renderbuffer_manager.cc [modify] https://crrev.com/23a5326c8ae2d90654faeda8cbd7225b08ed3412/gpu/command_buffer/service/renderbuffer_manager.h [modify] https://crrev.com/23a5326c8ae2d90654faeda8cbd7225b08ed3412/gpu/config/gpu_driver_bug_list.json [modify] https://crrev.com/23a5326c8ae2d90654faeda8cbd7225b08ed3412/gpu/config/gpu_driver_bug_workaround_type.h
,
Jan 27 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/fd06d67f6eaf0003f0a53e2635ccbbabbf56fb72 commit fd06d67f6eaf0003f0a53e2635ccbbabbf56fb72 Author: Kenneth Russell <kbr@chromium.org> Date: Sat Jan 27 05:21:57 2018 Restrict depth/stencil renderbuffer workaround on macOS. The driver bug's been fixed in 10.13.4. Bug: 775202 Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel Change-Id: Ie068990d23884956b614705aa9f7abf81953cec0 Reviewed-on: https://chromium-review.googlesource.com/889414 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Kai Ninomiya <kainino@chromium.org> Reviewed-by: Zhenyao Mo <zmo@chromium.org> Cr-Commit-Position: refs/heads/master@{#532160} [modify] https://crrev.com/fd06d67f6eaf0003f0a53e2635ccbbabbf56fb72/gpu/config/gpu_driver_bug_list.json |
||||||||||||||||||||
►
Sign in to add a comment |
||||||||||||||||||||
Comment 1 by kbr@chromium.org
, Oct 16 2017Cc: ericrk@chromium.org