New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 820478 link

Starred by 1 user
Status: Verified
Owner: ----
Closed: Oct 31
Cc:
pgangishetty@chromium.org
ka...@chromium.org
sontis@chromium.org
█ waihong@chromium.org
█ patricia@chromium.org
mcasas@chromium.org
dchan@chromium.org
█ marc...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

[Chameleon_HDMI] Several hosts are failing tests due to chrome crash causing RPC failures

Project Member Reported by matthewjoseph@chromium.org, Mar 9 2018 
This seems to have started recently, any builds > R67-10471.0.0.

RPC display.close_tab is throwing an error: DevToolsClientUrlError: [Errno 111] Connection refused

Full log of one failure:
03/08 22:42:14.129 DEBUG|remote_facade_fact:0141| RPC 'display.close_tab'(('0x7fe138b20d50',), {}) returns 'Traceback (most recent call last):\n  File "./multimedia_xmlrpc_server.py", line 92, in _dispatch\n    return func(*params)\n  File "/usr/local/autotest/cros/multimedia/display_facade_native.py", line 671, in close_tab\n    self.set_fullscreen(False)\n  File "/usr/local/autotest/cros/multimedia/display_facade_native.py", line 551, in set_fullscreen\n    extension = self._resource.get_extension()\n  File "/usr/local/autotest/common_lib/cros/retry.py", line 218, in func_retry\n    remaining_time)\n  File "/usr/local/autotest/common_lib/cros/retry.py", line 123, in timeout\n    default_result = func(*args, **kwargs)\n  File "/usr/local/autotest/cros/multimedia/facade_resource.py", line 208, in get_extension\n    extension = self._chrome.autotest_ext\n  File "/usr/local/autotest/common_lib/cros/chrome.py", line 220, in autotest_ext\n    return self.get_extension(self._autotest_ext_path)\n  File "/usr/local/autotest/common_lib/cros/chrome.py", line 213, in get_extension\n    return self.browser.extensions[ext]\n  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/browser/extension_dict.py", line 19, in __getitem__\n    return self.GetByExtensionId(load_extension.extension_id)[0]\n  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/browser/extension_dict.py", line 34, in GetByExtensionId\n    return self._extension_backend[extension_id]\n  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/extension_backend.py", line 31, in __getitem__\n    for context_id in self._extension_backend_list.IterContextIds():\n  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend_list.py", line 86, in IterContextIds\n    self._Update()\n  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend_list.py", line 99, in _Update\n    backends_map = self._devtools_client.GetUpdatedInspectableContexts()\n  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py", line 376, in GetUpdatedInspectableContexts\n    contexts = self._ListInspectableContexts()\n  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py", line 301, in _ListInspectableContexts\n    return self._devtools_http.RequestJson(\'\')\n  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_http.py", line 107, in RequestJson\n    return json.loads(self.Request(path, timeout))\n  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_http.py", line 87, in Request\n    self._conn.request(\'GET\', endpoint)\n  File "/usr/local/lib64/python2.7/httplib.py", line 1053, in request\n    self._send_request(method, url, body, headers)\n  File "/usr/local/lib64/python2.7/httplib.py", line 1093, in _send_request\n    self.endheaders(body)\n  File "/usr/local/lib64/python2.7/httplib.py", line 1049, in endheaders\n    self._send_output(message_body)\n  File "/usr/local/lib64/python2.7/httplib.py", line 893, in _send_output\n    self.send(msg)\n  File "/usr/local/lib64/python2.7/httplib.py", line 855, in send\n    self.connect()\n  File "/usr/local/lib64/python2.7/httplib.py", line 832, in connect\n    self.timeout, self.source_address)\n  File "/usr/local/lib64/python2.7/socket.py", line 575, in create_connection\n    raise err\nDevToolsClientUrlError: [Errno 111] Connection refused\n'.
03/08 22:42:14.130 ERROR|remote_facade_fact:0188| Failed RPC 'display.close_tab'(('0x7fe138b20d50',), {}) with status [<type 'exceptions.Exception'>].


Stainless link to failures:
https://stainless.corp.google.com/search?view=list&first_date=2018-03-01&last_date=2018-03-09&suite=chameleon_hdmi_perbuild&build=%5ER67&status=FAIL&reason=DevToolsClientUrlError&exclude_cts=false&exclude_not_run=false&exclude_non_release=false&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=true

Comment 1 by ka...@chromium.org, Mar 9 2018 

This failure is usually accompanied by browser crash, which terminates chrome, resp there is no tab to close at end of test.

I can see a crash dump in one of the test logs at https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/182149988-chromeos-test/chromeos2-row10-rack6-host5/debug/

Operating system: Linux
                  0.0.0 Linux 3.18.0-17019-g47869de17b79 #1 SMP PREEMPT Thu Mar 8 19:26:18 PST 2018 x86_64
CPU: amd64
     family 6 model 78 stepping 3
     1 CPU

GPU: UNKNOWN

Crash reason:  SIGSEGV
Crash address: 0x0
Process uptime: not available

Thread 0 (crashed)
 0  chrome!resource_coordinator::TabManager::PurgeBackgroundedTabsIfNeeded() [tab_manager.cc : 488 + 0xa]
    rax = 0x000035cdd4334000   rdx = 0x0000000000000006
    rcx = 0xffffca30c7699fd2   rbx = 0x000035cdd0a95000
    rsi = 0x000035cdd1349210   rdi = 0x000035cdd4334000
    rbp = 0x00007ffc2ad2bd80   rsp = 0x00007ffc2ad2bd30
     r8 = 0x0000000000000000    r9 = 0x0000000000000000
    r10 = 0x0000000000004000   r11 = 0x0000000000000000
    r12 = 0x00005d9e1cb60750   r13 = 0x0000000000000014
    r14 = 0x000035cdd4b4ef18   r15 = 0x000035cdd051064c
    rip = 0x00005d9e16f5560c
    Found by: given as instruction pointer in context
 1  chrome!base::Timer::RunScheduledTask() [callback.h : 124 + 0x3]
    rbx = 0x000035cdd0c04930   rbp = 0x00007ffc2ad2bdb0
    rsp = 0x00007ffc2ad2bd90   r12 = 0x00005d9e1cb60750
    r13 = 0x0000000000000000   r14 = 0x00007ffc2ad2bdd8
    r15 = 0x000035cdd051064c   rip = 0x00005d9e171c6f53
    Found by: call frame info
 2  chrome!base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) [callback.h : 95 + 0x3]
    rbx = 0x00007ffc2ad2bf20   rbp = 0x00007ffc2ad2be40
    rsp = 0x00007ffc2ad2bdc0   r12 = 0x00005d9e1cb60750
    r13 = 0x0000000000000000   r14 = 0x00007ffc2ad2bdd8
    r15 = 0x000035cdd051064c   rip = 0x00005d9e14f40fa4
    Found by: call frame info
 3  chrome!base::MessageLoop::RunTask(base::PendingTask*) [message_loop.cc : 395 + 0x8]
    rbx = 0x00005d9e16f54402   rbp = 0x00007ffc2ad2bf10
    rsp = 0x00007ffc2ad2be50   r12 = 0x000035cdd0510500
    r13 = 0x0000000000000000   r14 = 0x00007ffc2ad2bf20
    r15 = 0x00007ffc2ad2bf20   rip = 0x00005d9e14f4362c
    Found by: call frame info
 4  chrome!base::MessageLoop::DoDelayedWork(base::TimeTicks*) [message_loop.cc : 407 + 0x8]
    rbx = 0x0000000065a1c8df   rbp = 0x00007ffc2ad2c060
    rsp = 0x00007ffc2ad2bf20   r12 = 0x000035cdd05706d0
    r13 = 0x0000000000000000   r14 = 0x000035cdd05706d0
    r15 = 0x000035cdd0510500   rip = 0x00005d9e14f43c4f
    Found by: call frame info
 5  chrome!base::MessagePumpLibevent::Run(base::MessagePump::Delegate*) [message_pump_libevent.cc : 230 + 0xc]
    rbx = 0x000035cdd05706c0   rbp = 0x00007ffc2ad2c0d0
    rsp = 0x00007ffc2ad2c070   r12 = 0x000035cdd05706d0
    r13 = 0x0000000000000000   r14 = 0x000035cdd0510500
    r15 = 0x0000000000000000   rip = 0x00005d9e14f43f8b
    Found by: call frame info
 6  chrome!std::__1::unique_ptr<suggestions::ImageDecoderImpl::DecodeImageRequest, std::__1::default_delete<suggestions::ImageDecoderImpl::DecodeImageRequest> >::unique_ptr(std::__1::unique_ptr<suggestions::ImageDecoderImpl::DecodeImageRequest, std::__1::default_delete<suggestions::ImageDecoderImpl::DecodeImageRequest> >&&) [run_loop.cc : 133 + 0x9]
    rbx = 0x00007ffc2ad2c148   rbp = 0x00007ffc2ad2c0f0
    rsp = 0x00007ffc2ad2c0e0   r12 = 0x00005d9e1cc297b0
    r13 = 0x00007ffc2ad2c708   r14 = 0x000035cdd04e2c58
...

Comment 2 by ka...@chromium.org, Mar 9 2018

Cc: waihong@chromium.org
Components: OS>Kernel>Display
Labels: M-67
It looks like after image comparison for resolution 1440x900 the browser crashes

Can you see if anything can be reproed manually?



03/09 01:06:43.034 INFO |    screen_capture:0048| Capturing the screen on Chameleon...
03/09 01:06:44.914 INFO |    screen_capture:0106| Capturing the external screen on CrOS...
03/09 01:06:45.217 DEBUG|             utils:0282| [client] 01:06:44 INFO | Dispatching method <bound method DisplayFacadeNative.take_external_screenshot of <autotest_lib.client.cros.multimedia.display_facade_native.DisplayFacadeNative object at 0x79d5f1ad8210>> with args ('/tmp/tmpR564gz.png',)
03/09 01:06:45.217 DEBUG|             utils:0282| [client] 01:06:44 DEBUG| Running 'modetest -c'
03/09 01:06:45.218 DEBUG|             utils:0282| [client] 01:06:44 DEBUG| 'modetest -c'
03/09 01:06:45.218 DEBUG|             utils:0282| [client] 01:06:44 DEBUG| Running 'modetest -p'
03/09 01:06:45.218 DEBUG|             utils:0282| [client] 01:06:44 DEBUG| 'modetest -p'
03/09 01:06:45.218 DEBUG|             utils:0282| [client] 127.0.0.1 - - [09/Mar/2018 01:06:45] "POST /RPC2 HTTP/1.1" 200 -
03/09 01:06:45.218 DEBUG|remote_facade_fact:0141| RPC 'display.take_external_screenshot'(('/tmp/tmpR564gz.png',), {}) returns None.
03/09 01:06:45.219 DEBUG|      abstract_ssh:0413| get_file. source: /tmp/tmpR564gz.png, dest: /tmp/tmpR564gz.png, delete_dest: False,preserve_perm: True, preserve_symlinks:False
03/09 01:06:45.219 DEBUG|      abstract_ssh:0425| Using Rsync.
03/09 01:06:45.219 DEBUG|             utils:0214| Running 'rsync -L  --timeout=1800 --rsh='/usr/bin/ssh -a -x  -o Protocol=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22' -az --no-o --no-g  root@chromeos2-row10-rack6-host5:"/tmp/tmpR564gz.png" "/tmp/tmpR564gz.png"'
03/09 01:06:45.598 INFO | screen_comparison:0088| Comparing the images between Chameleon and CrOS-Ext...
03/09 01:06:45.680 INFO | screen_comparison:0110| Result: all pixels match (within +/- 1)
03/09 01:07:45.899 DEBUG|             utils:0282| [client] 01:06:45 INFO | Dispatching method <bound method DisplayFacadeNative.close_tab of <autotest_lib.client.cros.multimedia.display_facade_native.DisplayFacadeNative object at 0x79d5f1ad8210>> with args ('0x79d5f15efe10',)
03/09 01:07:45.899 DEBUG|             utils:0282| [client] 01:06:45 WARNI| <class 'telemetry.internal.backends.chrome_inspector.devtools_http.DevToolsClientConnectionError'>('')
03/09 01:07:45.899 DEBUG|             utils:0282| [client] 01:06:45 WARNI| Retrying in 1.054373 seconds...
03/09 01:07:45.899 DEBUG|             utils:0282| [client] 01:06:46 WARNI| <class 'telemetry.internal.backends.chrome_inspector.devtools_http.DevToolsClientUrlError'>([Errno 111] Connection refused)

....

03/09 01:07:45.908 DEBUG|             utils:0282| [client] 01:07:44 WARNI| Retrying in 1.195011 seconds...
03/09 01:07:45.909 DEBUG|             utils:0282| [client] 01:07:45 WARNI| <class 'telemetry.internal.backends.chrome_inspector.devtools_http.DevToolsClientUrlError'>([Errno 111] Connection refused)
03/09 01:07:45.909 DEBUG|             utils:0282| [client] 127.0.0.1 - - [09/Mar/2018 01:07:45] "POST /RPC2 HTTP/1.1" 200 -
03/09 01:07:45.910 DEBUG|remote_facade_fact:0141| RPC 'display.close_tab'(('0x79d5f15efe10',), {}) returns 'Traceback (most recent call last):\n  File "./multimedia_xmlrpc_server.py", line 92, in _dispatch\n    return func(*params)\n  File "/usr/local/autotest/cros/multimedia/display_facade_native.py", line 671, in close_tab\n    self.set_fullscreen(False)\n  File "/usr/local/autotest/cros/multimedia/display_facade_native.py", line 551, in set_fullscreen\n    extension = self._resource.get_extension()\n  File "/usr/local/autotest/common_lib/cros/retry.py", line 218, in func_retry\n    remaining_time)\n  File "/usr/local/autotest/common_lib/cros/retry.py", line 123, in timeout\n    default_result = func(*args, **kwargs)\n  File "/usr/local/autotest/cros/multimedia/facade_resource.py", line 208, in get_extension\n    extension = self._chrome.autotest_ext\n  File "/usr/local/autotest/common_lib/cros/chrome.py", line 220, in autotest_ext\n    return self.get_extension(self._autotest_ext_path)\n  File "/usr/local/autotest/common_lib/cros/chrome.py", line 213, in get_extension\n    return self.browser.extensions[ext]\n  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/browser/extension_dict.py", line 19, in __getitem__\n    return self.GetByExtensionId(load_extension.extension_id)[0]\n  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/browser/extension_dict.py", line 34, in GetByExtensionId\n    return self._extension_backend[extension_id]\n  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome/extension_backend.py", line 31, in __getitem__\n    for context_id in self._extension_backend_list.IterContextIds():\n  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend_list.py", line 86, in IterContextIds\n    self._Update()\n  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend_list.py", line 99, in _Update\n    backends_map = self._devtools_client.GetUpdatedInspectableContexts()\n  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py", line 376, in GetUpdatedInspectableContexts\n    contexts = self._ListInspectableContexts()\n  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py", line 301, in _ListInspectableContexts\n    return self._devtools_http.RequestJson(\'\')\n  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_http.py", line 107, in RequestJson\n    return json.loads(self.Request(path, timeout))\n  File "/usr/local/telemetry/src/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_http.py", line 87, in Request\n    self._conn.request(\'GET\', endpoint)\n  File "/usr/local/lib64/python2.7/httplib.py", line 1053, in request\n    self._send_request(method, url, body, headers)\n  File "/usr/local/lib64/python2.7/httplib.py", line 1093, in _send_request\n    self.endheaders(body)\n  File "/usr/local/lib64/python2.7/httplib.py", line 1049, in endheaders\n    self._send_output(message_body)\n  File "/usr/local/lib64/python2.7/httplib.py", line 893, in _send_output\n    self.send(msg)\n  File "/usr/local/lib64/python2.7/httplib.py", line 855, in send\n    self.connect()\n  File "/usr/local/lib64/python2.7/httplib.py", line 832, in connect\n    self.timeout, self.source_address)\n  File "/usr/local/lib64/python2.7/socket.py", line 575, in create_connection\n    raise err\nDevToolsClientUrlError: [Errno 111] Connection refused\n'.
03/09 01:07:45.911 ERROR|remote_facade_fact:0188| Failed RPC 'display.close_tab'(('0x79d5f15efe10',), {}) with status [<type 'exceptions.Exception'>].
...

Comment 3 by matthewjoseph@chromium.org, Mar 9 2018 

On kevin, 2-10-6-9

the failing RPC call is 
03/08 23:11:56.433 ERROR|remote_facade_fact:0188| Failed RPC 'display.set_fullscreen'((True,), {}) with status [<type 'exceptions.Exception'>].

On all the boards I looked at so far, there is a chrome crash in the debug/ folder.  However, it is not clear what step is causing the chrome crash.  Will try to reproduce manually.

Kevin log:
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/182103751-chromeos-test/chromeos2-row10-rack6-host9/debug/

Comment 4 by matthewjoseph@chromium.org, Mar 9 2018

Summary: [Chameleon_HDMI] Several hosts are failing tests due to chrome crash causing RPC failures (was: [Chameleon_HDMI] Several boards are failing tests due to RPC exception: 'display.close_tab' on chameleon )

Comment 5 by matthewjoseph@chromium.org, Mar 9 2018 

Cannot reproduce this chrome crash manually on kevin (R67-10473.0.0)

Comment 6 by matthewjoseph@chromium.org, Oct 31

Status: Verified (was: Untriaged)
This is no longer happening.  Closing.
►

Sign in to add a comment