Project: chromium Issues People Development process History Sign in
New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.
Starred by 3 users
Status: Fixed
Owner:
Closed: Apr 24
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 0
Type: Bug



Sign in to add a comment
frequent security_SandboxStatus WebSocketTimeoutException failures
Project Member Reported by nxia@chromium.org, Apr 20 Back to list
https://luci-milo.appspot.com/buildbot/chromeos/lumpy-paladin/28057


https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/113421973-chromeos-test/chromeos6-row2-rack7-host18/security_SandboxStatus/debug/


security_SandboxStatus failed: "The app is probably crashed"

*******************************************************************************
(/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:477 _ConvertExceptionFromInspectorWebsocket) The app is probably crashed:

Found Minidump: False
Stack Trace:
********************************************************************************
	Cannot get stack trace on CrOS
********************************************************************************
Standard output:
********************************************************************************
	Cannot get standard output on CrOS
********************************************************************************
System log:
(Not implemented)
********************************************************************************
(/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:482 _ConvertExceptionFromInspectorWebsocket) Original exception:
timed out
********************************************************************************
(/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:503 _AddDebuggingInformation) Received a socket error in the browser connection and the tab still exists. The operation probably timed out.
********************************************************************************
(/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:504 _AddDebuggingInformation) Debugger url: ws://127.0.0.1:49210/devtools/page/ccd00a6a-bd83-44ce-8400-114974def672. Exception log follows the after_iteration_hooks.
04/19 20:28:38.148 DEBUG|              test:0392| starting after_iteration_hooks
04/19 20:28:38.150 DEBUG|        base_utils:0185| Running 'logger "autotest finished iteration /usr/local/autotest/results/default/security_SandboxStatus/sysinfo/iteration.1"'
04/19 20:28:38.169 DEBUG|              test:0395| after_iteration_hooks completed
04/19 20:28:38.172 WARNI|              test:0615| The test failed with the following exception
Traceback (most recent call last):
  File "/usr/local/autotest/common_lib/test.py", line 609, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/usr/local/autotest/common_lib/test.py", line 823, in _call_test_function
    raise error.UnhandledTestFail(e)
UnhandledTestFail: Unhandled TimeoutException: 
********************************************************************************
(/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:477 _ConvertExceptionFromInspectorWebsocket) The app is probably crashed:

Found Minidump: False
Stack Trace:
********************************************************************************
	Cannot get stack trace on CrOS
********************************************************************************
Standard output:
********************************************************************************
	Cannot get standard output on CrOS
********************************************************************************
System log:
(Not implemented)
********************************************************************************
(/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:482 _ConvertExceptionFromInspectorWebsocket) Original exception:
timed out
********************************************************************************
(/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:503 _AddDebuggingInformation) Received a socket error in the browser connection and the tab still exists. The operation probably timed out.
********************************************************************************
(/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:504 _AddDebuggingInformation) Debugger url: ws://127.0.0.1:49210/devtools/page/ccd00a6a-bd83-44ce-8400-114974def672
Traceback (most recent call last):
  File "/usr/local/autotest/common_lib/test.py", line 817, in _call_test_function
    return func(*args, **dargs)
  File "/usr/local/autotest/common_lib/test.py", line 470, in execute
    dargs)
  File "/usr/local/autotest/common_lib/test.py", line 347, in _call_run_once_with_retry
    postprocess_profiled_run, args, dargs)
  File "/usr/local/autotest/common_lib/test.py", line 380, in _call_run_once
    self.run_once(*args, **dargs)
  File "/usr/local/autotest/tests/security_SandboxStatus/security_SandboxStatus.py", line 60, in run_once
    self._CheckAdequatelySandboxed()
  File "/usr/local/autotest/tests/security_SandboxStatus/security_SandboxStatus.py", line 39, in _CheckAdequatelySandboxed
    "document.getElementsByTagName('p')[0].textContent")
  File "/usr/local/autotest/tests/security_SandboxStatus/security_SandboxStatus.py", line 26, in _CheckSandboxPage
    self._tab.Navigate(url)
  File "/usr/local/telemetry/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/browser/web_contents.py", line 228, in Navigate
    self._inspector_backend.Navigate(url, script_to_evaluate_on_commit, timeout)
  File "/usr/local/telemetry/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 40, in inner
    inspector_backend._ConvertExceptionFromInspectorWebsocket(e)
  File "/usr/local/telemetry/src/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 37, in inner
    return func(inspector_backend, *args, **kwargs)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 176, in Navigate
    self._page.Navigate(url, script_to_evaluate_on_commit, timeout)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_page.py", line 125, in Navigate
    self.WaitForNavigate(timeout)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_page.py", line 94, in WaitForNavigate
    self._inspector_websocket.DispatchNotifications(remaining_time)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 134, in DispatchNotifications
    self._Receive(timeout)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 149, in _Receive
    data = self._socket.recv()
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 293, in recv
    opcode, data = self.recv_data()
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 310, in recv_data
    opcode, frame = self.recv_data_frame(control_frame)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 323, in recv_data_frame
    frame = self.recv_frame()
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 357, in recv_frame
    return self.frame_buffer.recv_frame()
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 336, in recv_frame
    self.recv_header()
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 286, in recv_header
    header = self.recv_strict(2)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 371, in recv_strict
    bytes_ = self.recv(min(16384, shortage))
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 427, in _recv
    return recv(self.sock, bufsize)
  File "/usr/local/telemetry/src/third_party/catapult/telemetry/third_party/websocket-client/websocket/_socket.py", line 83, in recv
    raise WebSocketTimeoutException(message)
TimeoutException: 
********************************************************************************
(/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:477 _ConvertExceptionFromInspectorWebsocket) The app is probably crashed:

Found Minidump: False
Stack Trace:
********************************************************************************
	Cannot get stack trace on CrOS
********************************************************************************
Standard output:
********************************************************************************
	Cannot get standard output on CrOS
********************************************************************************
System log:
(Not implemented)
********************************************************************************
(/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:482 _ConvertExceptionFromInspectorWebsocket) Original exception:
timed out
********************************************************************************
(/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:503 _AddDebuggingInformation) Received a socket error in the browser connection and the tab still exists. The operation probably timed out.
********************************************************************************
(/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:504 _AddDebuggingInformation) Debugger url: ws://127.0.0.1:49210/devtools/page/ccd00a6a-bd83-44ce-8400-114974def672

 
Cc: x...@chromium.org ihf@chromium.org
+ gardeners, is this chrome crash or chromeos crash?
Cc: -ihf@chromium.org achuith@chromium.org
Owner: nxia@chromium.org
I am sure you just resurrected issue 706939 by pinning Chrome to 3064.

And this is really not my problem.
Cc: akes...@chromium.org jrbarnette@chromium.org
security_SandboxStatus timeout is the only failure in master-paladin/14349.

It looks like pinning chrome version from 3072 to 3064 has reduced the chrome crashes and save CQs from crbug.com/713004. But 3064 contains the issue 706939, which has happened consistently on different paladins in 3 CQ runs. 

What would be the good chrome version? will pinning back to 3065 help? If so, I need to fix the chrome_pin tool.

https://luci-milo.appspot.com/buildbot/chromeos/veyron_mighty-paladin/5056

https://luci-milo.appspot.com/buildbot/chromeos/master-paladin/14349
I created an unpin CL at:

https://chromium-review.googlesource.com/c/482999/


Cc: snanda@chromium.org
Project Member Comment 7 by bugdroid1@chromium.org, Apr 20
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/79f4d41b16e359453676bb62605ac546adc22804

commit 79f4d41b16e359453676bb62605ac546adc22804
Author: Ningning Xia <nxia@google.com>
Date: Thu Apr 20 18:09:14 2017

Chrome: Unpin chrome

DO NOT REVERT THIS CL.
In general, reverting chrome (un)pin CLs does not do what you expect.
Instead, use `cros pinchrome` to generate new CLs.

BUG= chromium:713531 
TEST=None

Change-Id: I83f6330a462a8178ec7357c375bbab40e42e158e
Reviewed-on: https://chromium-review.googlesource.com/482999
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Tested-by: Ningning Xia <nxia@chromium.org>

[delete] https://crrev.com/3570f2f9e27d148d08e1e65bc1645a69e1287a0e/profiles/default/linux/package.mask/chromepin

Labels: -Pri-2 Pri-0
Owner: x...@chromium.org
Summary: frequent security_SandboxStatus WebSocketTimeoutException failures (was: lumpy-paladin: security_SandboxStatus timeout)
Cc: steve...@chromium.org ihf@chromium.org
I see that Achuith recently moved this test out of bvt-inline temporarily (and returned it on Apr 11). I suggest we do this again. Preparing CL
Maybe no need. Those paladins are still using chromeos-chrome-59.0.3064.0_rc-r1. which is a known broken version according to comment#4.

So how long will it take for the unpin chrome take effect? It has been 10 hours.
The unpin is already in effect, in the sense that the chrome pfq is now allowed to uprev chrome. But getting the chrome pfq to pass and do so is another matter. So we might be stuck on 3064 for a while.
Project Member Comment 15 by bugdroid1@chromium.org, Apr 21
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/551c10669608ff2aaf270273026043d143fa1c56

commit 551c10669608ff2aaf270273026043d143fa1c56
Author: Aviv Keshet <akeshet@chromium.org>
Date: Fri Apr 21 06:10:56 2017

temporarily move security_SandboxStatus out of bvt-inline

BUG= chromium:713531 
TEST=None

Change-Id: I8f1defe9d7628565a9d6ce8ea4ab433215eb18a9
Reviewed-on: https://chromium-review.googlesource.com/483943
Commit-Queue: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
Trybot-Ready: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>

[modify] https://crrev.com/551c10669608ff2aaf270273026043d143fa1c56/client/site_tests/security_SandboxStatus/control

I'm a little disappointed that we keep blaming the security_SandboxStatus test, causing us to lose critical test coverage (the Chrome sandbox is pretty critical) for a *known* bug in Chrome, which has nothing to do with the test. We know that Chrome has a fix. We could have pinned Chrome to a version that had the Chrome fix, for example, stabilize the CQ, and then unpin Chrome.
This is a short term measure while we find a new version of Chrome to rev or pin to that shows neither this Sandbox problem nor the double-free  problem (crbug.com/713968) that's taking down the lab network (crbug.com/708679)
I generally agree with you, but I don't think the deputy knew of a better chrome version to pin to at the time, to try to resolve 708679
Issue 706939 has been merged into this issue.
Project Member Comment 20 by bugdroid1@chromium.org, Apr 22
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/885017e92f0936b001f04167374c32821c953de9

commit 885017e92f0936b001f04167374c32821c953de9
Author: Aviv Keshet <akeshet@chromium.org>
Date: Sat Apr 22 04:57:53 2017

Revert "temporarily move security_SandboxStatus out of bvt-inline"

Chrome was manually uprevved, so we should be passed this issue.

This reverts commit 551c10669608ff2aaf270273026043d143fa1c56.

BUG= chromium:713531 ,  714170 
TEST=None

Change-Id: Icd405271e76a134cf2568d2bc546573e4109dfcd
Reviewed-on: https://chromium-review.googlesource.com/483968
Commit-Ready: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Steven Bennetts <stevenjb@chromium.org>
Reviewed-by: Jorge Lucangeli Obes <jorgelo@chromium.org>

[modify] https://crrev.com/885017e92f0936b001f04167374c32821c953de9/client/site_tests/security_SandboxStatus/control

Status: Fixed
I think this issue can be safely closed.
Labels: VerifyIn-60
Labels: VerifyIn-61
Sign in to add a comment