WebglConformance flaky on Nexus 5X GPU.FYI |
||||||||
Issue descriptionFailed in build 365. Not sure if it's the same problem as issue 609252 , so opening a new one. [ RUN ] WebglConformance.conformance_extensions_oes_element_index_uint (INFO) 2016-06-23 16:00:51,529 cmd_helper._ValidateAndLogCommand:153 [host]> /tmp/isolated_rundEJAvy/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00e1775f893b38ea shell '( rm -f /data/local/chrome-trace-config.json );echo %$?' (INFO) 2016-06-23 16:00:51,639 cache_temperature.EnsurePageCacheTemperature:55 PageCacheTemperature: any (INFO) 2016-06-23 16:02:21,925 cmd_helper._ValidateAndLogCommand:153 [host]> /tmp/isolated_rundEJAvy/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00e1775f893b38ea shell '( uiautomator dump /data/local/tmp/temp_file-d31fd0887741b );echo %$?' (INFO) 2016-06-23 16:02:24,276 cmd_helper._ValidateAndLogCommand:153 [host]> /tmp/isolated_rundEJAvy/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00e1775f893b38ea pull /data/local/tmp/temp_file-d31fd0887741b /tmp/isolated_tmpZt8NJT/tmpojKPAE/tmp_ReadFileWithPull (INFO) 2016-06-23 16:02:24,414 cmd_helper._ValidateAndLogCommand:153 [host]> /tmp/isolated_rundEJAvy/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00e1775f893b38ea shell 'rm -f /data/local/tmp/temp_file-d31fd0887741b' (INFO) 2016-06-23 16:02:24,434 cmd_helper._ValidateAndLogCommand:153 [host]> /tmp/isolated_rundEJAvy/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00e1775f893b38ea shell '( logcat -d -t 500 );echo %$?' (ERROR) 2016-06-23 16:02:24,599 exceptions.__init__:71 Problem when trying to gather standard output: 'NoneType' object has no attribute 'splitlines' (INFO) 2016-06-23 16:02:34,617 cmd_helper._ValidateAndLogCommand:153 [host]> /tmp/isolated_rundEJAvy/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00e1775f893b38ea shell '( uiautomator dump /data/local/tmp/temp_file-32d8049cfa148 );echo %$?' (INFO) 2016-06-23 16:02:36,418 cmd_helper._ValidateAndLogCommand:153 [host]> /tmp/isolated_rundEJAvy/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00e1775f893b38ea pull /data/local/tmp/temp_file-32d8049cfa148 /tmp/isolated_tmpZt8NJT/tmpiuWFDO/tmp_ReadFileWithPull (INFO) 2016-06-23 16:02:36,553 cmd_helper._ValidateAndLogCommand:153 [host]> /tmp/isolated_rundEJAvy/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00e1775f893b38ea shell 'rm -f /data/local/tmp/temp_file-32d8049cfa148' (INFO) 2016-06-23 16:02:36,558 cmd_helper._ValidateAndLogCommand:153 [host]> /tmp/isolated_rundEJAvy/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00e1775f893b38ea shell '( logcat -d -t 500 );echo %$?' (ERROR) 2016-06-23 16:02:36,692 exceptions.__init__:71 Problem when trying to gather standard output: 'NoneType' object has no attribute 'splitlines' (INFO) 2016-06-23 16:02:36,698 browser.DumpStateUponFailure:326 *************** BROWSER STANDARD OUTPUT *************** (INFO) 2016-06-23 16:02:36,698 browser.DumpStateUponFailure:328 None (INFO) 2016-06-23 16:02:36,698 browser.DumpStateUponFailure:331 *********** END OF BROWSER STANDARD OUTPUT ************ (INFO) 2016-06-23 16:02:36,698 browser.DumpStateUponFailure:333 ********************* BROWSER LOG ********************* (ERROR) 2016-06-23 16:02:36,699 browser.DumpStateUponFailure:337 Failed to get browser log: Traceback (most recent call last): File "/tmp/isolated_rundEJAvy/third_party/catapult/telemetry/telemetry/internal/browser/browser.py", line 335, in DumpStateUponFailure logging.info(self.GetLogFileContents()) File "/tmp/isolated_rundEJAvy/third_party/catapult/telemetry/telemetry/internal/browser/browser.py", line 276, in GetLogFileContents return self._browser_backend.GetLogFileContents() File "/tmp/isolated_rundEJAvy/third_party/catapult/telemetry/telemetry/internal/backends/browser_backend.py", line 43, in GetLogFileContents with file(self.log_file_path) as f: TypeError: coercing to Unicode: need string or buffer, NoneType found (INFO) 2016-06-23 16:02:36,699 browser.DumpStateUponFailure:338 ***************** END OF BROWSER LOG ****************** (WARNING) 2016-06-23 16:02:36,699 shared_page_state.DumpStateUponFailure:144 Taking screenshots upon failures disabled. Traceback (most recent call last): File "/tmp/isolated_rundEJAvy/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 85, in _RunStoryAndProcessErrorIfNeeded state.RunStory(results) File "/tmp/isolated_rundEJAvy/content/test/gpu/gpu_tests/gpu_test_base.py", line 122, in RunStory RunStoryWithRetries(DesktopGpuSharedPageState, self, results) File "/tmp/isolated_rundEJAvy/content/test/gpu/gpu_tests/gpu_test_base.py", line 72, in RunStoryWithRetries super(cls, shared_page_state).RunStory(results) File "/tmp/isolated_rundEJAvy/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 319, in RunStory self._current_page.Run(self) File "/tmp/isolated_rundEJAvy/third_party/catapult/telemetry/telemetry/page/__init__.py", line 95, in Run shared_state.page_test.RunNavigateSteps(self, current_tab) File "/tmp/isolated_rundEJAvy/third_party/catapult/telemetry/telemetry/page/legacy_page_test.py", line 191, in RunNavigateSteps page.RunNavigateSteps(action_runner) File "/tmp/isolated_rundEJAvy/content/test/gpu/gpu_tests/webgl_conformance.py", line 199, in RunNavigateSteps super(WebglConformancePage, self).RunNavigateSteps(action_runner) File "/tmp/isolated_rundEJAvy/third_party/catapult/telemetry/telemetry/page/__init__.py", line 104, in RunNavigateSteps url, script_to_evaluate_on_commit=self.script_to_evaluate_on_commit) File "/tmp/isolated_rundEJAvy/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 121, in Navigate timeout_in_seconds=timeout_in_seconds)) File "/tmp/isolated_rundEJAvy/third_party/catapult/telemetry/telemetry/internal/actions/action_runner.py", line 44, in _RunAction action.RunAction(self._tab) File "/tmp/isolated_rundEJAvy/third_party/catapult/telemetry/telemetry/internal/actions/navigate.py", line 29, in RunAction time_left_in_seconds) File "/tmp/isolated_rundEJAvy/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 82, in WaitForDocumentReadyStateToBeInteractiveOrBetter 'document.readyState == "complete"', timeout) File "/tmp/isolated_rundEJAvy/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 136, in WaitForJavaScriptExpression e.message + '\n' + debug_message) TimeoutException: Exception thrown when trying to capture console output: TimeoutException('',) (INFO) 2016-06-23 16:04:06,789 cmd_helper._ValidateAndLogCommand:153 [host]> /tmp/isolated_rundEJAvy/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00e1775f893b38ea shell '( uiautomator dump /data/local/tmp/temp_file-504296ffea3df );echo %$?' (INFO) 2016-06-23 16:04:08,650 cmd_helper._ValidateAndLogCommand:153 [host]> /tmp/isolated_rundEJAvy/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00e1775f893b38ea pull /data/local/tmp/temp_file-504296ffea3df /tmp/isolated_tmpZt8NJT/tmpZTLZTW/tmp_ReadFileWithPull (INFO) 2016-06-23 16:04:08,831 cmd_helper._ValidateAndLogCommand:153 [host]> /tmp/isolated_rundEJAvy/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00e1775f893b38ea shell 'rm -f /data/local/tmp/temp_file-504296ffea3df' (INFO) 2016-06-23 16:04:08,864 cmd_helper._ValidateAndLogCommand:153 [host]> /tmp/isolated_rundEJAvy/third_party/catapult/devil/bin/deps/linux2/x86_64/bin/adb -s 00e1775f893b38ea shell '( logcat -d -t 500 );echo %$?' (ERROR) 2016-06-23 16:04:08,998 exceptions.__init__:71 Problem when trying to gather standard output: 'NoneType' object has no attribute 'splitlines' [ FAILED ] WebglConformance.conformance_extensions_oes_element_index_uint (197712 ms)
,
Jun 27 2016
Now it's WebglConformance.conformance_extensions_oes_element_index_uint in build 457. Similar symptoms. I have no idea where is the problem: catapult / devil / telemetry? Ken, can you please route this?
,
Jun 27 2016
Link to failing build: https://build.chromium.org/p/chromium.gpu.fyi/builders/Android%20Release%20%28Nexus%205X%29/builds/457 Full stdout, and an excerpt, attached. It looks like the GPU process is crashing during the previous test (WebglConformance.conformance_extensions_oes_texture_half_float_with_video), or some other failure happens which causes a lost context. This test then times out. The next test, WebglConformance.conformance_extensions_oes_element_index_uint, then times out because the browser's not in a good state to run the next test. I think switching over the webgl_conformance_tests to the new gpu_integration_test harness will solve this problem, because the new harness restarts the browser after flaky and unexpected test failures. I'll work on this next.
,
Jun 27 2016
,
Jun 30 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/6ab8b3bc2f12e0d4a2857f303bbbae1395c37fd2 commit 6ab8b3bc2f12e0d4a2857f303bbbae1395c37fd2 Author: kbr <kbr@chromium.org> Date: Thu Jun 30 19:08:32 2016 Switch WebGL 1.0 conformance tests to new GPU integration test harness. This will allow these tests to be sharded and improve flaky test handling. BUG= 352807 , 622948 CQ_INCLUDE_TRYBOTS=tryserver.chromium.linux:linux_optional_gpu_tests_rel;tryserver.chromium.mac:mac_optional_gpu_tests_rel;tryserver.chromium.win:win_optional_gpu_tests_rel Review-Url: https://codereview.chromium.org/2110803004 Cr-Commit-Position: refs/heads/master@{#403240} [modify] https://crrev.com/6ab8b3bc2f12e0d4a2857f303bbbae1395c37fd2/content/test/gpu/generate_buildbot_json.py [modify] https://crrev.com/6ab8b3bc2f12e0d4a2857f303bbbae1395c37fd2/testing/buildbot/chromium.gpu.fyi.json [modify] https://crrev.com/6ab8b3bc2f12e0d4a2857f303bbbae1395c37fd2/testing/buildbot/chromium.gpu.json
,
Jul 2 2016
ynovikov@ pointed out in https://codereview.chromium.org/2110803004/ that it looks like that CL broke the WebGL conformance tests running on Android: https://build.chromium.org/p/chromium.gpu.fyi/builders/Android%20Release%20%28Nexus%205%29/builds/401 Excerpt: ====================================================================== ERROR: setUpClass (gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/tmp/isolated_runXvNNCa/content/test/gpu/gpu_tests/webgl_conformance_integration_test.py", line 75, in setUpClass cls.CustomizeOptions() File "/tmp/isolated_runXvNNCa/content/test/gpu/gpu_tests/webgl_conformance_integration_test.py", line 60, in CustomizeOptions validator.CustomizeBrowserOptions(cls._finder_options.browser_options) File "/tmp/isolated_runXvNNCa/content/test/gpu/gpu_tests/webgl_conformance.py", line 110, in CustomizeBrowserOptions if (browser.target_os.startswith('android') and AttributeError: 'NoneType' object has no attribute 'target_os' Reverting.
,
Jul 2 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0cc54a485ca313d7a56aa5f978d1247cc974aa98 commit 0cc54a485ca313d7a56aa5f978d1247cc974aa98 Author: kbr <kbr@chromium.org> Date: Sat Jul 02 04:37:48 2016 Revert of Switch WebGL 1.0 conformance tests to new GPU integration test harness. (patchset #1 id:1 of https://codereview.chromium.org/2110803004/ ) Reason for revert: Broke Android bots on chromium.gpu.fyi per ynovikov@'s comment above; see http://crbug.com/622948 . Original issue's description: > Switch WebGL 1.0 conformance tests to new GPU integration test harness. > > This will allow these tests to be sharded and improve flaky test > handling. > > BUG= 352807 , 622948 > CQ_INCLUDE_TRYBOTS=tryserver.chromium.linux:linux_optional_gpu_tests_rel;tryserver.chromium.mac:mac_optional_gpu_tests_rel;tryserver.chromium.win:win_optional_gpu_tests_rel > > Committed: https://crrev.com/6ab8b3bc2f12e0d4a2857f303bbbae1395c37fd2 > Cr-Commit-Position: refs/heads/master@{#403240} TBR=zmo@chromium.org,jmadill@chromium.org # Not skipping CQ checks because original CL landed more than 1 days ago. BUG= 352807 , 622948 Review-Url: https://codereview.chromium.org/2120043002 Cr-Commit-Position: refs/heads/master@{#403604} [modify] https://crrev.com/0cc54a485ca313d7a56aa5f978d1247cc974aa98/content/test/gpu/generate_buildbot_json.py [modify] https://crrev.com/0cc54a485ca313d7a56aa5f978d1247cc974aa98/testing/buildbot/chromium.gpu.fyi.json [modify] https://crrev.com/0cc54a485ca313d7a56aa5f978d1247cc974aa98/testing/buildbot/chromium.gpu.json
,
Jul 7 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/14b4d7e08c4ff427b6db64d795c1768f860cbb56 commit 14b4d7e08c4ff427b6db64d795c1768f860cbb56 Author: catapult-deps-roller <catapult-deps-roller@chromium.org> Date: Thu Jul 07 04:30:07 2016 Roll src/third_party/catapult/ 9f8a0a8c0..71bab5766 (15 commits). https://chromium.googlesource.com/external/github.com/catapult-project/catapult.git/+log/9f8a0a8c0d45..71bab5766221 $ git log 9f8a0a8c0..71bab5766 --date=short --no-merges --format='%ad %ae %s' BUG= 352807 , 622948 , 625064 ,472699, 605842 ,472699 TBR=catapult-sheriff@chromium.org Review-Url: https://codereview.chromium.org/2125883005 Cr-Commit-Position: refs/heads/master@{#404074} [modify] https://crrev.com/14b4d7e08c4ff427b6db64d795c1768f860cbb56/DEPS
,
Jul 8 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/ec09e72eef18ba20616f95fd82d9ca084809d935 commit ec09e72eef18ba20616f95fd82d9ca084809d935 Author: kbr <kbr@chromium.org> Date: Fri Jul 08 14:27:45 2016 Switch WebGL 1.0 conformance tests to new GPU integration test harness. (Re-land after fixing new test harness on Android.) This will allow these tests to be sharded and improve flaky test handling. BUG= 352807 , 622948 CQ_INCLUDE_TRYBOTS=tryserver.chromium.linux:linux_optional_gpu_tests_rel;tryserver.chromium.mac:mac_optional_gpu_tests_rel;tryserver.chromium.win:win_optional_gpu_tests_rel TBR=zmo@chromium.org, jmadill@chromium.org Review-Url: https://codereview.chromium.org/2126163003 Cr-Commit-Position: refs/heads/master@{#404377} [modify] https://crrev.com/ec09e72eef18ba20616f95fd82d9ca084809d935/content/test/gpu/generate_buildbot_json.py [modify] https://crrev.com/ec09e72eef18ba20616f95fd82d9ca084809d935/testing/buildbot/chromium.gpu.fyi.json [modify] https://crrev.com/ec09e72eef18ba20616f95fd82d9ca084809d935/testing/buildbot/chromium.gpu.json
,
Jul 9 2016
At this point I see one recent failure on this bot: https://build.chromium.org/p/chromium.gpu.fyi/builders/Android%20Release%20%28Nexus%205X%29/builds/822 WebglConformance_conformance_extensions_oes_vertex_array_object failed, but that doesn't seem to be what really happened. Rather, the earlier test, WebglConformance_conformance_extensions_oes_texture_half_float_with_video, timed out with the following error: (warning) http://127.0.0.1:45225/conformance/extensions/oes-texture-half-float-with-video.html:0: GL_INVALID_OPERATION : glGenSyncTokenCHROMIUM: fence sync must be flushed before generating sync token (warning) http://127.0.0.1:45225/conformance/extensions/oes-texture-half-float-with-video.html:0: WebGL: CONTEXT_LOST_WEBGL: loseContext: context lost This test needs to be fully debugged, gotten to work, and then de-flaked. I'm unassigning myself from this task. Need help from others at this point.
,
Jan 9 2017
,
Jan 9 2017
,
Aug 10 2017
With the current set of suppressions this bot is pretty stable now: https://luci-milo.appspot.com/buildbot/chromium.gpu.fyi/Android%20Release%20%28Nexus%205X%29/?limit=200 There are a few infrastructure-level failures but on the whole webgl_conformance_tests are running reliably. Archiving. |
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by ynovikov@chromium.org
, Jun 24 2016