DevTools' Screenshot functionality timing out intermittently on Mac |
||||||||||||
Issue descriptionThe screenshot_sync_tests have become flaky on Mac OS apparently due to DevTools' screenshot command timing out. When the timeout happens it looks like subsequent tests fail. Attached are stdout from these two runs (in case the logs expire): https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/214584 https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_chromium_rel_ng/builds/214490 These tests are failing on fairly recent Mac Minis, and usually run in about 25 seconds, so there is no excuse for them timing out at the 1 minute mark. These tests also run on the CQ, so figuring out what is going wrong is especially important. They cover crucial browser functionality and ensure that pixels reach the screen correctly.
,
Apr 21 2016
,
Apr 21 2016
,
Apr 21 2016
Unfortunately running this test is so slow on my local machine that I can't produce a repro case. Filed Issue 605739 about that.
,
Apr 23 2016
nednguyen@ helped me get numpy and cv2 into a local build of Telemetry. With the attached changes I tried running the test in a loop (after building the Release browser) ./content/test/gpu/run_gpu_test.py screenshot_sync --browser=release --pageset-repeat=100 --max-failures=1 I wasn't able to easily provoke the failures. Maybe it happens more often on a slow machine, or in cold start situations.
,
Apr 23 2016
Pavel, do you think someone from the DevTools team could try running this test next week on Mac OS and see if they can provoke the failure seen on the Mac bots?
,
Apr 23 2016
Note that the tests flaking above were: ScreenshotSync.WithCanvas ScreenshotSync.WithDivs
,
Apr 23 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c8d7fb1437ad351bae551f5a7aaaf736e83a07ac commit c8d7fb1437ad351bae551f5a7aaaf736e83a07ac Author: kbr <kbr@chromium.org> Date: Sat Apr 23 03:41:53 2016 Suppress screenshot_sync_tests flakiness on Mac with Intel GPU. Update the bug ID for the preexisting Windows suppression. BUG= 599776 , 605699 CQ_INCLUDE_TRYBOTS=tryserver.chromium.linux:linux_optional_gpu_tests_rel;tryserver.chromium.mac:mac_optional_gpu_tests_rel;tryserver.chromium.win:win_optional_gpu_tests_rel NOTRY=true TBR=zmo@chromium.org Review URL: https://codereview.chromium.org/1919623002 Cr-Commit-Position: refs/heads/master@{#389356} [modify] https://crrev.com/c8d7fb1437ad351bae551f5a7aaaf736e83a07ac/content/test/gpu/gpu_tests/screenshot_sync_expectations.py
,
Apr 25 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c8d7fb1437ad351bae551f5a7aaaf736e83a07ac commit c8d7fb1437ad351bae551f5a7aaaf736e83a07ac Author: kbr <kbr@chromium.org> Date: Sat Apr 23 03:41:53 2016 Suppress screenshot_sync_tests flakiness on Mac with Intel GPU. Update the bug ID for the preexisting Windows suppression. BUG= 599776 , 605699 CQ_INCLUDE_TRYBOTS=tryserver.chromium.linux:linux_optional_gpu_tests_rel;tryserver.chromium.mac:mac_optional_gpu_tests_rel;tryserver.chromium.win:win_optional_gpu_tests_rel NOTRY=true TBR=zmo@chromium.org Review URL: https://codereview.chromium.org/1919623002 Cr-Commit-Position: refs/heads/master@{#389356} [modify] https://crrev.com/c8d7fb1437ad351bae551f5a7aaaf736e83a07ac/content/test/gpu/gpu_tests/screenshot_sync_expectations.py
,
May 2 2016
,
May 2 2016
To be clear, no root cause for this bug was found. Changing status to WontFix. |
||||||||||||
►
Sign in to add a comment |
||||||||||||
Comment 1 by kbr@chromium.org
, Apr 21 2016Log excerpts for easier searching: [ RUN ] ScreenshotSync.WithCanvas ... [1284:49223:0420/131003:WARNING:simple_synchronous_entry.cc(929)] Could not open platform files for entry. Can't get standard output with --show-stdout Traceback (most recent call last): File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 84, in _RunStoryAndProcessErrorIfNeeded state.RunStory(results) File "/b/swarm_slave/work/isolated/runG4__0c/content/test/gpu/gpu_tests/gpu_test_base.py", line 111, in RunStory RunStoryWithRetries(GpuSharedPageState, self, results) File "/b/swarm_slave/work/isolated/runG4__0c/content/test/gpu/gpu_tests/gpu_test_base.py", line 72, in RunStoryWithRetries super(cls, shared_page_state).RunStory(results) File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 326, in RunStory self._current_page, self._current_tab, results) File "/b/swarm_slave/work/isolated/runG4__0c/content/test/gpu/gpu_tests/screenshot_sync.py", line 57, in ValidateAndMeasurePage CheckScreenshot() File "/b/swarm_slave/work/isolated/runG4__0c/content/test/gpu/gpu_tests/screenshot_sync.py", line 46, in CheckScreenshot screenshot = tab.Screenshot(5) File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/browser/tab.py", line 117, in Screenshot return self._inspector_backend.Screenshot(timeout) File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 35, in inner inspector_backend._ConvertExceptionFromInspectorWebsocket(e) File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 32, in inner return func(inspector_backend, *args, **kwargs) File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 139, in Screenshot return self._page.CaptureScreenshot(timeout) File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_page.py", line 146, in CaptureScreenshot res = self._inspector_websocket.SyncRequest(request, timeout) File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 110, in SyncRequest res = self._Receive(timeout) File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 149, in _Receive data = self._socket.recv() File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/third_party/websocket-client/websocket.py", line 596, in recv opcode, data = self.recv_data() File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/third_party/websocket-client/websocket.py", line 606, in recv_data frame = self.recv_frame() File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/third_party/websocket-client/websocket.py", line 637, in recv_frame self._frame_header = self._recv_strict(2) File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/third_party/websocket-client/websocket.py", line 746, in _recv_strict bytes = self._recv(shortage) File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/third_party/websocket-client/websocket.py", line 732, in _recv raise WebSocketTimeoutException(e.message) TimeoutException: ******************************************************************************** (/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:315 _ConvertExceptionFromInspectorWebsocket) The app is probably crashed: Found Minidump: False Stack Trace: ******************************************************************************** No crash dump found. ******************************************************************************** Standard output: ******************************************************************************** ******************************************************************************** ******************************************************************************** (/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:320 _ConvertExceptionFromInspectorWebsocket) Original exception: timed out ******************************************************************************** (/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:341 _AddDebuggingInformation) Received a socket error in the browser connection and the tab no longer exists. The tab probably crashed. ******************************************************************************** (/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:342 _AddDebuggingInformation) Debugger url: ws://127.0.0.1:49332/devtools/page/1F3EF9C0-17E7-4ED0-8432-925BAE98E011 [ FAILED ] ScreenshotSync.WithCanvas (67528 ms) [ RUN ] ScreenshotSync.WithDivs Traceback (most recent call last): File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 77, in _RunStoryAndProcessErrorIfNeeded state.WillRunStory(story) File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 253, in WillRunStory if len(self.browser.tabs) == 0: File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/browser/tab_list.py", line 15, in __len__ return self._tab_list_backend.__len__() File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend_list.py", line 95, in __len__ self._Update() File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend_list.py", line 99, in _Update backends_map = self._devtools_client.GetUpdatedInspectableContexts() File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py", line 290, in GetUpdatedInspectableContexts contexts = self._ListInspectableContexts() File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py", line 215, in _ListInspectableContexts return self._devtools_http.RequestJson('') File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_http.py", line 107, in RequestJson return json.loads(self.Request(path, timeout)) File "/b/swarm_slave/work/isolated/runG4__0c/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_http.py", line 88, in Request response = self._conn.getresponse() File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1132, in getresponse response.begin() File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 453, in begin version, status, reason = self._read_status() File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 409, in _read_status line = self.fp.readline(_MAXLINE + 1) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 480, in readline data = self._sock.recv(self._rbufsize) DevToolsClientConnectionError: timed out (WARNING) 2016-04-20 13:11:44,079 desktop_browser_backend.Close:527 Failed to gracefully shutdown. (WARNING) 2016-04-20 13:11:44,079 desktop_browser_backend.Close:531 Proceed to kill the browser. [ FAILED ] ScreenshotSync.WithDivs