frequent security_SandboxStatus WebSocketTimeoutException failures |
||||||||||
Issue descriptionhttps://luci-milo.appspot.com/buildbot/chromeos/lumpy-paladin/28057 https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/113421973-chromeos-test/chromeos6-row2-rack7-host18/security_SandboxStatus/debug/ security_SandboxStatus failed: "The app is probably crashed" ******************************************************************************* (/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:477 _ConvertExceptionFromInspectorWebsocket) The app is probably crashed: Found Minidump: False Stack Trace: ******************************************************************************** Cannot get stack trace on CrOS ******************************************************************************** Standard output: ******************************************************************************** Cannot get standard output on CrOS ******************************************************************************** System log: (Not implemented) ******************************************************************************** (/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:482 _ConvertExceptionFromInspectorWebsocket) Original exception: timed out ******************************************************************************** (/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:503 _AddDebuggingInformation) Received a socket error in the browser connection and the tab still exists. The operation probably timed out. ******************************************************************************** (/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:504 _AddDebuggingInformation) Debugger url: ws://127.0.0.1:49210/devtools/page/ccd00a6a-bd83-44ce-8400-114974def672. Exception log follows the after_iteration_hooks. 04/19 20:28:38.148 DEBUG| test:0392| starting after_iteration_hooks 04/19 20:28:38.150 DEBUG| base_utils:0185| Running 'logger "autotest finished iteration /usr/local/autotest/results/default/security_SandboxStatus/sysinfo/iteration.1"' 04/19 20:28:38.169 DEBUG| test:0395| after_iteration_hooks completed 04/19 20:28:38.172 WARNI| test:0615| The test failed with the following exception Traceback (most recent call last): File "/usr/local/autotest/common_lib/test.py", line 609, in _exec _call_test_function(self.execute, *p_args, **p_dargs) File "/usr/local/autotest/common_lib/test.py", line 823, in _call_test_function raise error.UnhandledTestFail(e) UnhandledTestFail: Unhandled TimeoutException: ******************************************************************************** (/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:477 _ConvertExceptionFromInspectorWebsocket) The app is probably crashed: Found Minidump: False Stack Trace: ******************************************************************************** Cannot get stack trace on CrOS ******************************************************************************** Standard output: ******************************************************************************** Cannot get standard output on CrOS ******************************************************************************** System log: (Not implemented) ******************************************************************************** (/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:482 _ConvertExceptionFromInspectorWebsocket) Original exception: timed out ******************************************************************************** (/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:503 _AddDebuggingInformation) Received a socket error in the browser connection and the tab still exists. The operation probably timed out. ******************************************************************************** (/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:504 _AddDebuggingInformation) Debugger url: ws://127.0.0.1:49210/devtools/page/ccd00a6a-bd83-44ce-8400-114974def672 Traceback (most recent call last): File "/usr/local/autotest/common_lib/test.py", line 817, in _call_test_function return func(*args, **dargs) File "/usr/local/autotest/common_lib/test.py", line 470, in execute dargs) File "/usr/local/autotest/common_lib/test.py", line 347, in _call_run_once_with_retry postprocess_profiled_run, args, dargs) File "/usr/local/autotest/common_lib/test.py", line 380, in _call_run_once self.run_once(*args, **dargs) File "/usr/local/autotest/tests/security_SandboxStatus/security_SandboxStatus.py", line 60, in run_once self._CheckAdequatelySandboxed() File "/usr/local/autotest/tests/security_SandboxStatus/security_SandboxStatus.py", line 39, in _CheckAdequatelySandboxed "document.getElementsByTagName('p')[0].textContent") File "/usr/local/autotest/tests/security_SandboxStatus/security_SandboxStatus.py", line 26, in _CheckSandboxPage self._tab.Navigate(url) File "/usr/local/telemetry/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function return func(*args, **kwargs) File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 228, in Navigate self._inspector_backend.Navigate(url, script_to_evaluate_on_commit, timeout) File "/usr/local/telemetry/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function return func(*args, **kwargs) File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 40, in inner inspector_backend._ConvertExceptionFromInspectorWebsocket(e) File "/usr/local/telemetry/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function return func(*args, **kwargs) File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 37, in inner return func(inspector_backend, *args, **kwargs) File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 176, in Navigate self._page.Navigate(url, script_to_evaluate_on_commit, timeout) File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_page.py", line 125, in Navigate self.WaitForNavigate(timeout) File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_page.py", line 94, in WaitForNavigate self._inspector_websocket.DispatchNotifications(remaining_time) File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 134, in DispatchNotifications self._Receive(timeout) File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 149, in _Receive data = self._socket.recv() File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 293, in recv opcode, data = self.recv_data() File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 310, in recv_data opcode, frame = self.recv_data_frame(control_frame) File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 323, in recv_data_frame frame = self.recv_frame() File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 357, in recv_frame return self.frame_buffer.recv_frame() File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 336, in recv_frame self.recv_header() File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 286, in recv_header header = self.recv_strict(2) File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 371, in recv_strict bytes_ = self.recv(min(16384, shortage)) File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 427, in _recv return recv(self.sock, bufsize) File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_socket.py", line 83, in recv raise WebSocketTimeoutException(message) TimeoutException: ******************************************************************************** (/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:477 _ConvertExceptionFromInspectorWebsocket) The app is probably crashed: Found Minidump: False Stack Trace: ******************************************************************************** Cannot get stack trace on CrOS ******************************************************************************** Standard output: ******************************************************************************** Cannot get standard output on CrOS ******************************************************************************** System log: (Not implemented) ******************************************************************************** (/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:482 _ConvertExceptionFromInspectorWebsocket) Original exception: timed out ******************************************************************************** (/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:503 _AddDebuggingInformation) Received a socket error in the browser connection and the tab still exists. The operation probably timed out. ******************************************************************************** (/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:504 _AddDebuggingInformation) Debugger url: ws://127.0.0.1:49210/devtools/page/ccd00a6a-bd83-44ce-8400-114974def672
,
Apr 20 2017
I am sure you just resurrected issue 706939 by pinning Chrome to 3064. And this is really not my problem.
,
Apr 20 2017
another example: https://luci-milo.appspot.com/buildbot/chromeos/wolf-paladin/14099
,
Apr 20 2017
security_SandboxStatus timeout is the only failure in master-paladin/14349. It looks like pinning chrome version from 3072 to 3064 has reduced the chrome crashes and save CQs from crbug.com/713004. But 3064 contains the issue 706939, which has happened consistently on different paladins in 3 CQ runs. What would be the good chrome version? will pinning back to 3065 help? If so, I need to fix the chrome_pin tool. https://luci-milo.appspot.com/buildbot/chromeos/veyron_mighty-paladin/5056 https://luci-milo.appspot.com/buildbot/chromeos/master-paladin/14349
,
Apr 20 2017
I created an unpin CL at: https://chromium-review.googlesource.com/c/482999/
,
Apr 20 2017
,
Apr 20 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/79f4d41b16e359453676bb62605ac546adc22804 commit 79f4d41b16e359453676bb62605ac546adc22804 Author: Ningning Xia <nxia@google.com> Date: Thu Apr 20 18:09:14 2017 Chrome: Unpin chrome DO NOT REVERT THIS CL. In general, reverting chrome (un)pin CLs does not do what you expect. Instead, use `cros pinchrome` to generate new CLs. BUG= chromium:713531 TEST=None Change-Id: I83f6330a462a8178ec7357c375bbab40e42e158e Reviewed-on: https://chromium-review.googlesource.com/482999 Reviewed-by: Aviv Keshet <akeshet@chromium.org> Tested-by: Ningning Xia <nxia@chromium.org> [delete] https://crrev.com/3570f2f9e27d148d08e1e65bc1645a69e1287a0e/profiles/default/linux/package.mask/chromepin
,
Apr 21 2017
Still see this issue in many paladins: https://uberchromegw.corp.google.com/i/chromeos/builders/veyron_mighty-paladin/builds/5062 https://uberchromegw.corp.google.com/i/chromeos/builders/link-paladin/builds/28099 https://uberchromegw.corp.google.com/i/chromeos/builders/lumpy-paladin/builds/28065 https://uberchromegw.corp.google.com/i/chromeos/builders/wolf-tot-paladin/builds/9797
,
Apr 21 2017
,
Apr 21 2017
,
Apr 21 2017
I see that Achuith recently moved this test out of bvt-inline temporarily (and returned it on Apr 11). I suggest we do this again. Preparing CL
,
Apr 21 2017
https://chromium-review.googlesource.com/#/c/483943/ (context: achuith did this recently in https://chromium-review.googlesource.com/475107 and https://chromium-review.googlesource.com/468347 so I am jumping off the same bridge)
,
Apr 21 2017
Maybe no need. Those paladins are still using chromeos-chrome-59.0.3064.0_rc-r1. which is a known broken version according to comment#4. So how long will it take for the unpin chrome take effect? It has been 10 hours.
,
Apr 21 2017
The unpin is already in effect, in the sense that the chrome pfq is now allowed to uprev chrome. But getting the chrome pfq to pass and do so is another matter. So we might be stuck on 3064 for a while.
,
Apr 21 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/551c10669608ff2aaf270273026043d143fa1c56 commit 551c10669608ff2aaf270273026043d143fa1c56 Author: Aviv Keshet <akeshet@chromium.org> Date: Fri Apr 21 06:10:56 2017 temporarily move security_SandboxStatus out of bvt-inline BUG= chromium:713531 TEST=None Change-Id: I8f1defe9d7628565a9d6ce8ea4ab433215eb18a9 Reviewed-on: https://chromium-review.googlesource.com/483943 Commit-Queue: Aviv Keshet <akeshet@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org> Trybot-Ready: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Ilja H. Friedel <ihf@chromium.org> [modify] https://crrev.com/551c10669608ff2aaf270273026043d143fa1c56/client/site_tests/security_SandboxStatus/control
,
Apr 21 2017
I'm a little disappointed that we keep blaming the security_SandboxStatus test, causing us to lose critical test coverage (the Chrome sandbox is pretty critical) for a *known* bug in Chrome, which has nothing to do with the test. We know that Chrome has a fix. We could have pinned Chrome to a version that had the Chrome fix, for example, stabilize the CQ, and then unpin Chrome.
,
Apr 21 2017
This is a short term measure while we find a new version of Chrome to rev or pin to that shows neither this Sandbox problem nor the double-free problem (crbug.com/713968) that's taking down the lab network (crbug.com/708679)
,
Apr 21 2017
I generally agree with you, but I don't think the deputy knew of a better chrome version to pin to at the time, to try to resolve 708679
,
Apr 21 2017
Issue 706939 has been merged into this issue.
,
Apr 22 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/885017e92f0936b001f04167374c32821c953de9 commit 885017e92f0936b001f04167374c32821c953de9 Author: Aviv Keshet <akeshet@chromium.org> Date: Sat Apr 22 04:57:53 2017 Revert "temporarily move security_SandboxStatus out of bvt-inline" Chrome was manually uprevved, so we should be passed this issue. This reverts commit 551c10669608ff2aaf270273026043d143fa1c56. BUG= chromium:713531 , 714170 TEST=None Change-Id: Icd405271e76a134cf2568d2bc546573e4109dfcd Reviewed-on: https://chromium-review.googlesource.com/483968 Commit-Ready: Aviv Keshet <akeshet@chromium.org> Tested-by: Aviv Keshet <akeshet@chromium.org> Reviewed-by: Steven Bennetts <stevenjb@chromium.org> Reviewed-by: Jorge Lucangeli Obes <jorgelo@chromium.org> [modify] https://crrev.com/885017e92f0936b001f04167374c32821c953de9/client/site_tests/security_SandboxStatus/control
,
Apr 24 2017
I think this issue can be safely closed.
,
May 30 2017
,
Aug 1 2017
,
Jan 22 2018
|
||||||||||
►
Sign in to add a comment |
||||||||||
Comment 1 by nxia@chromium.org
, Apr 20 2017