WebglConformance_deqp_functional_gles3_shaderoperator_common_functions flaky timeout on Mac FYI Release (Intel) and AMD |
|||||||||||||
Issue descriptionhttps://ci.chromium.org/buildbot/chromium.gpu.fyi/Mac%20FYI%20Release%20%28Intel%29/730 https://ci.chromium.org/buildbot/chromium.gpu.fyi/Mac%20FYI%20Release%20%28Intel%29/617 [102/131] gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest.WebglConformance_deqp_functional_gles3_shaderoperator_common_functions failed unexpectedly 319.9650s: Traceback (most recent call last): _RunGpuTest at content/test/gpu/gpu_tests/gpu_integration_test.py:132 self.RunActualGpuTest(url, *args) RunActualGpuTest at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:203 getattr(self, test_name)(test_path, *args[1:]) _RunConformanceTest at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:217 self._CheckTestCompletion() _CheckTestCompletion at content/test/gpu/gpu_tests/webgl_conformance_integration_test.py:211 'webglTestHarness._finished', timeout=self._GetTestTimeout()) traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52 return func(*args, **kwargs) WaitForJavaScriptCondition at third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py:261 return self._tab.WaitForJavaScriptCondition(*args, **kwargs) traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52 return func(*args, **kwargs) WaitForJavaScriptCondition at third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py:239 return self._inspector_backend.WaitForJavaScriptCondition(*args, **kwargs) traced_function at third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52 return func(*args, **kwargs) WaitForJavaScriptCondition at third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:302 e.message + '\n' + debug_message) TimeoutException: Timed out while waiting 300s for IsJavaScriptExpressionTrue. Console output: Locals: IsJavaScriptExpressionTrue : <function IsJavaScriptExpressionTrue at 0x10c26caa0> condition : 'webglTestHarness._finished' context_id : None debug_message : 'Console output:\n' e : TimeoutException('Timed out while waiting 300s for IsJavaScriptExpressionTrue.',) kwargs : {} timeout : 300 Takes a reasonable time when passes, like in https://ci.chromium.org/buildbot/chromium.gpu.fyi/Mac%20FYI%20Release%20%28Intel%29/729: [102/131] gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest.WebglConformance_deqp_functional_gles3_shaderoperator_common_functions passed 183.7981s Will suppress.
,
Mar 16 2018
kbr@ can you please triage this? Thanks!
,
Mar 16 2018
,
Mar 21 2018
We should try to change the harness to capture and print out all of the test's console logs in the case the test times out, to figure out what part of the test is actually causing the timeout. There are a lot of sub-tests in this individual test.
,
May 30 2018
,
May 30 2018
This test is still timing out flakily: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Release%20%28Intel%29/3320 [106/135] gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest.WebglConformance_deqp_functional_gles3_shaderoperator_common_functions failed unexpectedly 968.6528s: Dashboard link: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=webgl2_conformance_tests&tests=gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest.WebglConformance_deqp_functional_gles3_shaderoperator_common_functions Flakes on ANGLE CQ: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_angle_rel_ng/1266 I think we should mark it as fail or even skip.
,
May 31 2018
,
May 31 2018
Thanks for marking this failing in https://chromium-review.googlesource.com/1079793 . Note that in Issue 846951 we may finally understand the cause of some random timeouts in these and other WebGL conformance tests.
,
May 31 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/6b9dc426fe3b26dba05886f450e51db6786ab855 commit 6b9dc426fe3b26dba05886f450e51db6786ab855 Author: Jamie Madill <jmadill@chromium.org> Date: Thu May 31 02:30:56 2018 Upgrade WebGL Mac failure to fail. The flaky expectation for shaderoperator_common_functions was insufficient and the test was still flakily failing. See issue for more details. Bug: 820225 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel Change-Id: Ie30e47e8ffaa44ebade9723742c541f3571a3aee Tbr: kbr@chromium.org Reviewed-on: https://chromium-review.googlesource.com/1079793 Reviewed-by: Jamie Madill <jmadill@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org> Cr-Commit-Position: refs/heads/master@{#563123} [modify] https://crrev.com/6b9dc426fe3b26dba05886f450e51db6786ab855/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py
,
Jun 9 2018
Jamie: do you think you could try removing this suppression now that the root failure in Issue 846951 has been addressed? Right now we have ~200 green builds on these bots: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Release%20%28Intel%29?limit=200 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Retina%20Release%20%28AMD%29?limit=200 and if the shaderoperator_common_functions failure was caused by Issue 846951 , we should turn the test back on. Thanks.
,
Jun 11 2018
Sure, I can try lifting the suppression.
,
Jun 11 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/53e7fc6f9fe8b214219c2980e10a254ebe1ed949 commit 53e7fc6f9fe8b214219c2980e10a254ebe1ed949 Author: Jamie Madill <jmadill@chromium.org> Date: Mon Jun 11 19:43:24 2018 Lift WebGL Mac test suppression. Tentatively fixed. Test: deqp/functional/gles3/shaderoperator/common_functions Bug: 820225 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel Change-Id: If66fee7d2beb1fecb9f9056425ee1dceb2a2acc1 Tbr: kbr@chromium.org Reviewed-on: https://chromium-review.googlesource.com/1095333 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org> Cr-Commit-Position: refs/heads/master@{#566110} [modify] https://crrev.com/53e7fc6f9fe8b214219c2980e10a254ebe1ed949/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py
,
Jun 12 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e3bf8d0fa3d1cff016fbf1034c3ba622e57d836a commit e3bf8d0fa3d1cff016fbf1034c3ba622e57d836a Author: Jamie Madill <jmadill@chromium.org> Date: Tue Jun 12 15:35:30 2018 Revert "Lift WebGL Mac test suppression." This reverts commit 53e7fc6f9fe8b214219c2980e10a254ebe1ed949. Reason for revert: Immediately started flaking: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Release%20%28Intel%29/3698 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Release%20%28Intel%29/3707 Original change's description: > Lift WebGL Mac test suppression. > > Tentatively fixed. > > Test: deqp/functional/gles3/shaderoperator/common_functions > Bug: 820225 > Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel > Change-Id: If66fee7d2beb1fecb9f9056425ee1dceb2a2acc1 > Tbr: kbr@chromium.org > Reviewed-on: https://chromium-review.googlesource.com/1095333 > Reviewed-by: Jamie Madill <jmadill@chromium.org> > Commit-Queue: Jamie Madill <jmadill@chromium.org> > Cr-Commit-Position: refs/heads/master@{#566110} TBR=jmadill@chromium.org,kbr@chromium.org Change-Id: Ia475bb6812aa31494339004bbfc85c663133d094 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 820225 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel Reviewed-on: https://chromium-review.googlesource.com/1097435 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org> Cr-Commit-Position: refs/heads/master@{#566431} [modify] https://crrev.com/e3bf8d0fa3d1cff016fbf1034c3ba622e57d836a/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py
,
Jun 26 2018
Thanks Jamie for trying and sorry for the trouble. Looking at a passing build: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Release%20%28Intel%29/4073 https://chromium-swarm.appspot.com/task?id=3e52aa38b1a4b810&refresh=10&show_raw=1 WebglConformance_deqp_functional_gles3_shaderoperator_common_functions was expected to fail, but passed. [107/134] gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest.WebglConformance_deqp_functional_gles3_shaderoperator_common_functions passed 226.2001s The test looks very slow so potentially we should just split it up so it's well under the threshold.
,
Jun 28 2018
Also seeing a timeout on ATI GPU on Mac Retina https://ci.chromium.org/p/chromium/builders/luci.chromium.try/mac_optional_gpu_tests_rel/5741 Going to expand the suppression.
,
Jun 28 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a07a77d23a425a6736318d553b9f924158b68efe commit a07a77d23a425a6736318d553b9f924158b68efe Author: Yuly Novikov <ynovikov@chromium.org> Date: Thu Jun 28 21:15:07 2018 Suppress a flaky timing out WebGL2 test deqp/functional/gles3/shaderoperator/common_functions.html on Mac AMD TBR=kbr@chromium.org Bug: 820225 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel Change-Id: I0515e950a90cda632ee52070dfa3c743ff19de2d Reviewed-on: https://chromium-review.googlesource.com/1119094 Commit-Queue: Yuly Novikov <ynovikov@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Reviewed-by: Yuly Novikov <ynovikov@chromium.org> Cr-Commit-Position: refs/heads/master@{#571261} [modify] https://crrev.com/a07a77d23a425a6736318d553b9f924158b68efe/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py
,
Jul 27
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/84424c67e64bede48014cd1f187cb32b48d314f3 commit 84424c67e64bede48014cd1f187cb32b48d314f3 Author: Kenneth Russell <kbr@chromium.org> Date: Fri Jul 27 23:49:06 2018 Update shaderoperator_common_functions suppression. It should be applied on all macOS versions, not just Sierra. This affects the following WebGL 2.0 conformance test: deqp/functional/gles3/shaderoperator/common_functions.html Tbr: kainino@chromium.org Bug: 820225 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel Change-Id: Ib0ff46c871d8ab74e56026aff96cd150e61a33d9 Reviewed-on: https://chromium-review.googlesource.com/1152452 Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#578864} [modify] https://crrev.com/84424c67e64bede48014cd1f187cb32b48d314f3/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py
,
Aug 28
Attention would need to be paid to this test to try splitting it up in order to make more progress. Unassigning and downgrading.
,
Oct 22
I saw this flaking on Win NVIDIA in https://ci.chromium.org/p/chromium/builders/luci.chromium.try/win_optional_gpu_tests_rel/9570 Look like normally it runs in 270 sec, which is close to timing out. Can splitting this test take a higher priority, as this flakes on CQ bot? And marking it flake probably won't be much help, since normal time is so close to timing out.
,
Oct 22
Let me try splitting up this test.
,
Oct 23
https://github.com/KhronosGroup/WebGL/pull/2733 is up for review splitting up this test. The expectations will have to be updated in order to roll forward WebGL conformance afterward, since the test's name has changed. I'll take care of that.
,
Oct 24
Rolling forward WebGL conformance in https://chromium-review.googlesource.com/1297570 . Will attempt to un-suppress these failures after this roll lands.
,
Oct 24
https://chromium-review.googlesource.com/1297570 landed (forgot to link to this bug); attempting to remove the Mac suppressions for these tests in https://chromium-review.googlesource.com/1298432 .
,
Oct 25
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/bfe1eaac68801495b0331985f264066633c20dfa commit bfe1eaac68801495b0331985f264066633c20dfa Author: Kenneth Russell <kbr@chromium.org> Date: Thu Oct 25 01:01:45 2018 Try to remove Mac shaderoperator/common_functions suppressions. The tests have been split up to fall well under the timeout threshold. Re-enabling to see whether they are now reliable. Tbr: ynovikov@chromium.org Bug: 820225 Change-Id: I73986640e500d2e69f94acb7c119425131724865 Reviewed-on: https://chromium-review.googlesource.com/c/1298432 Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#602551} [modify] https://crrev.com/bfe1eaac68801495b0331985f264066633c20dfa/content/test/gpu/gpu_tests/webgl2_conformance_expectations.py
,
Oct 30
The split-up of shaderoperator/common_functions seems to have worked. |
|||||||||||||
►
Sign in to add a comment |
|||||||||||||
Comment 1 by bugdroid1@chromium.org
, Mar 8 2018