Issue metadata
Sign in to add a comment
|
WebglConformance_conformance2_textures_misc_tex_mipmap_levels and misc_tex_base_level_bug flaky on 10.13.6 |
||||||||||||||||||||
Issue descriptionThe following two WebGL conformance tests have suddenly become flaky on 10.13.6. This indicates a recent OpenGL graphics driver bug introduced by Apple, likely around the management of texture objects' base levels. This state may be leaking across texture objects or OpenGL contexts. Failures are seen on this bot: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29?limit=200 Failing builds and failure modes: WebglConformance_conformance2_textures_misc_tex_base_level_bug https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3786 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3781 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3778 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3775 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3774 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3770 [41/141] gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest.WebglConformance_conformance2_textures_misc_tex_base_level_bug failed unexpectedly 1.6016s: Traceback (most recent call last): _RunGpuTest at content/test/gpu/gpu_tests/gpu_integration_test.py:138 self.RunActualGpuTest(url, *args) RunActualGpuTest at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:186 getattr(self, test_name)(test_path, *args[1:]) _RunConformanceTest at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:200 self._CheckTestCompletion() _CheckTestCompletion at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:196 self.fail(self._WebGLTestMessages(self.tab)) fail at .swarming_module/lib/python2.7/unittest/case.py:410 raise self.failureException(msg) AssertionError: should draw with [255, 0, 0, 255] at (0, 0) expected: 255,0,0,255 was 0,0,0,255 FAIL should draw with [255, 0, 0, 255] at (0, 0) expected: 255,0,0,255 was 0,0,0,255 WebglConformance_conformance2_textures_misc_tex_mipmap_levels https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3816 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3811 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3809 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3804 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3800 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3782 [55/142] gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest.WebglConformance_conformance2_textures_misc_tex_mipmap_levels failed unexpectedly 1.5130s: Traceback (most recent call last): _RunGpuTest at content/test/gpu/gpu_tests/gpu_integration_test.py:138 self.RunActualGpuTest(url, *args) RunActualGpuTest at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:186 getattr(self, test_name)(test_path, *args[1:]) _RunConformanceTest at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:200 self._CheckTestCompletion() _CheckTestCompletion at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:196 self.fail(self._WebGLTestMessages(self.tab)) fail at .swarming_module/lib/python2.7/unittest/case.py:410 raise self.failureException(msg) AssertionError: should draw with [255, 0, 0, 255] at (0, 0) expected: 255,0,0,255 was 0,0,0,255 FAIL should draw with [255, 0, 0, 255] at (0, 0) expected: 255,0,0,255 was 0,0,0,255 We need to file a Radar with Apple about these failures. They're causing other failures like random pixel test failures of Pixel_CSSFilterEffects: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3815 (Pixel_WebGLSadCanvas is also observed flaky but this is due to a preexisting bug in the drawing of the sad canvas which is on my plate to fix.)
,
Aug 7
Looks like it's an AMD-specific issue. The Intel devices aren't having any problems https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Release%20%28Intel%29
,
Aug 8
Ken, were you going to suppress the flakiness?
,
Aug 8
,
Aug 9
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/25ed0ad36dfd29d69a7ad090a450214e4cddffc5 commit 25ed0ad36dfd29d69a7ad090a450214e4cddffc5 Author: Kenneth Russell <kbr@chromium.org> Date: Thu Aug 09 23:41:24 2018 Suppress flaky tests on macOS 10.13.6 with AMD GPUs. WebGL conformance suppressions: conformance2/textures/misc/tex-mipmap-levels.html conformance2/textures/misc/tex-base-level-bug.html Pixel test suppressions: Pixel_CSSFilterEffects Pixel_WebGLSadCanvas Tbr: ynovikov@chromium.org Bug: 870856, 872423 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel Change-Id: I1420fe20b7a64bb40f3ba0209fd9bb08b3213b14 Reviewed-on: https://chromium-review.googlesource.com/1168470 Reviewed-by: Kenneth Russell <kbr@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Commit-Queue: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#581958} [modify] https://crrev.com/25ed0ad36dfd29d69a7ad090a450214e4cddffc5/content/test/gpu/gpu_tests/pixel_expectations.py [modify] https://crrev.com/25ed0ad36dfd29d69a7ad090a450214e4cddffc5/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py
,
Aug 10
After the suppression, shard #9 started to time out. https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3960 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3959 Looks like we need to either Fail or Skip WebglConformance_deqp_functional_gles3_shaderindexing_tmp Also, I saw some flakiness in WebglConformance_deqp_functional_gles3_shaderloop_* and WebglConformance_deqp_functional_gles3_shadermatrix_* and WebglConformance_deqp_functional_gles3_shaderoperator_* even before the suppression CL. For example, https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3958 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3957 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3956 Ken, could you please take care of suppressing whatever else do you think needs suppression here?
,
Aug 10
The symptom isn't just shard #9 timing out; random tests started to fail. Triaging some of these failures: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3963 WebglConformance_deqp_functional_gles3_shaderloop_while WebglConformance_deqp_functional_gles3_shadermatrix_div_dynamic https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3962 WebglConformance_deqp_functional_gles3_shaderoperator_unary_operator_02 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3961 WebglConformance_deqp_functional_gles3_shadermatrix_mul_const_mediump WebglConformance_deqp_functional_gles3_shaderoperator_binary_operator_10 WebglConformance_deqp_functional_gles3_shaderoperator_binary_operator_13 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3960 shard #9 timeout https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3949 WebglConformance_deqp_functional_gles3_shadermatrix_mul_const_mediump WebglConformance_deqp_functional_gles3_shaderoperator_binary_operator_10 WebglConformance_deqp_functional_gles3_shaderoperator_binary_operator_13 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3950 WebglConformance_deqp_functional_gles3_shadermatrix_div_const WebglConformance_deqp_functional_gles3_shadermatrix_mul_uniform_highp WebglConformance_deqp_functional_gles3_shadermatrix_mul_uniform_mediump WebglConformance_deqp_functional_gles3_shaderoperator_binary_operator_01 WebglConformance_deqp_functional_gles3_shaderoperator_binary_operator_06 Examining the 10.13.6 AMD MacBook Pros: https://chromium-swarm.appspot.com/botlist?c=id&c=os&c=task&c=status&f=gpu%3A1002%3A6821&f=hidpi%3A1&f=os%3AMac-10.13.6&f=cpu%3Ax86-64&f=pool%3AChrome-GPU&l=100&s=id%3Aasc It's clear that all of these failures are coming from two machines: build485-m4 and build700-m4. The others are OK. https://chromium-swarm.appspot.com/bot?id=build485-m4&sort_stats=total%3Adesc https://chromium-swarm.appspot.com/bot?id=build700-m4&sort_stats=total%3Adesc They might have the lock screen up, or something similar. build700-m4 seems to have crashed completely. I don't think additional suppressions are needed, though the one for WebglConformance_conformance2_textures_misc_tex_mipmap_levels has to be expanded. It's still failing occasionally despite the flaky retries: https://chromium-swarm.appspot.com/task?id=3f3ed7f6c0b8f110&refresh=10&show_raw=1 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/3964
,
Aug 12
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d6179b49f9ac994d10e56e18e35c551879ecc4be commit d6179b49f9ac994d10e56e18e35c551879ecc4be Author: Kenneth Russell <kbr@chromium.org> Date: Sun Aug 12 23:23:18 2018 Expand tex-mipmap-levels suppression. conformance2/textures/misc/tex-mipmap-levels.html fails three times in a row when it fails. Mark it as Fail rather than Flaky. Bug: 870856 Tbr: ynovikov@chromium.org Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel Change-Id: Ib49cb54ee451b4f5f556a0b5899c12d9e88d80b3 Reviewed-on: https://chromium-review.googlesource.com/1171942 Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#582483} [modify] https://crrev.com/d6179b49f9ac994d10e56e18e35c551879ecc4be/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py
,
Aug 20
,
Aug 20
,
Aug 20
WebglConformance_conformance2_textures_misc_tex_mipmap_levels observed flaky on Mac AMD with 10.12 after the above change to the suppression. Example: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20Pro%20FYI%20Release%20%28AMD%29/2140 Continued flakiness of WebglConformance_conformance2_textures_misc_tex_base_level_bug observed on 10.13.6, necessitating marking this test failing there: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/4170 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/4168 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/4165 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29/4163 Updating suppressions again in https://chromium-review.googlesource.com/1181655 .
,
Aug 20
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/9bd4e7a6acba9f18a01e616380b914aaea6aeb46 commit 9bd4e7a6acba9f18a01e616380b914aaea6aeb46 Author: Kenneth Russell <kbr@chromium.org> Date: Mon Aug 20 21:23:20 2018 Expand tex-mipmap-levels and tex-base-level-bug suppressions. Mark: conformance2/textures/misc/tex-mipmap-levels.html failing on 10.12 with AMD GPUs. The previous failure suppression covered this configuration, and it turns out it's unacceptably flaky there, too. Mark: conformance2/textures/misc/tex-base-level-bug.html failing on 10.13 with AMD GPUs. When the test fails, it fails three times in a row. Tbr: cblume@chromium.org Bug: 870856 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel Change-Id: Ibe020fabc74661dc4c8d8553a292ca5f8666ea77 Reviewed-on: https://chromium-review.googlesource.com/1181655 Reviewed-by: Kenneth Russell <kbr@chromium.org> Reviewed-by: Chris Blume <cblume@chromium.org> Cr-Commit-Position: refs/heads/master@{#584565} [modify] https://crrev.com/9bd4e7a6acba9f18a01e616380b914aaea6aeb46/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py
,
Aug 24
,
Aug 24
Issue 876240 has been merged into this issue.
,
Aug 24
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/d61f31a6fcb99ffcabd6987c0683451794ea7487 commit d61f31a6fcb99ffcabd6987c0683451794ea7487 Author: Kenneth Russell <kbr@chromium.org> Date: Fri Aug 24 20:18:08 2018 Suppress tex-base-level-bug on Mac AMD on 10.12 too. conformance2/textures/misc/tex-base-level-bug.html was unacceptably flaky back in that release as well. Tbr: kainino@chromium.org No-Try: True Bug: 870856 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel Change-Id: I7dd4859c118f92ae4f93f17da1a9020e1215dd54 Reviewed-on: https://chromium-review.googlesource.com/1188882 Commit-Queue: Kenneth Russell <kbr@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#585960} [modify] https://crrev.com/d61f31a6fcb99ffcabd6987c0683451794ea7487/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py
,
Aug 24
This seems fixed at this point. These two bots seem acceptably green now: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Retina%20Release%20%28AMD%29?limit=200 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Release%20%28Intel%29?limit=200
,
Nov 7
Still awaiting fixes from Apple.
,
Nov 27
,
Nov 27
|
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by kbr@chromium.org
, Aug 3