New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 715607 link

Starred by 1 user

Issue metadata

Status: WontFix
Owner:
OOO until 2019-01-24
Closed: Apr 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: ----
Type: ----



Sign in to add a comment

smoothness.tough_canvas_cases failing on chromium.perf/Linux Perf

Project Member Reported by sullivan@chromium.org, Apr 26 2017

Issue description

smoothness.tough_canvas_cases failing on chromium.perf/Linux Perf

failing on chrome/test/data/perf/canvas_bench/many_images.html

failure range: 464556:464655
https://chromium.googlesource.com/chromium/src/+log/e49d7dfd68361fcdd6b83bb696fc475276afae8f%5E..94309c21892ee97c409b3f7dc509e414dd9095a9?pretty=fuller

Builders failed on: 
- Linux Perf: 
  https://build.chromium.org/p/chromium.perf/builders/Linux%20Perf

https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Fchromium.perf%2FLinux_Perf%2F604%2F%2B%2Frecipes%2Fsteps%2Fsmoothness.tough_canvas_cases_on__102b__GPU_on_Linux%2F0%2Fstdout

(INFO) 2017-04-26 00:34:02,776 browser.DumpStateUponFailure:365  *************** BROWSER STANDARD OUTPUT ***************
(INFO) 2017-04-26 00:34:02,776 browser.DumpStateUponFailure:367  [22001:22001:0426/003054.348627:ERROR:sandbox_linux.cc(344)] InitializeSandbox() called with multiple threads in process gpu-process.

(INFO) 2017-04-26 00:34:02,776 browser.DumpStateUponFailure:370  *********** END OF BROWSER STANDARD OUTPUT ************
(INFO) 2017-04-26 00:34:02,776 browser.DumpStateUponFailure:372  ********************* BROWSER LOG *********************
(INFO) 2017-04-26 00:34:02,776 browser.DumpStateUponFailure:374  No log file
(INFO) 2017-04-26 00:34:02,776 browser.DumpStateUponFailure:377  ***************** END OF BROWSER LOG ******************
(WARNING) 2017-04-26 00:35:02,837 desktop_browser_backend._GetAllCrashpadMinidumps:366  No path to crashpad_database_util found
(INFO) 2017-04-26 00:35:02,837 desktop_browser_backend._GetMostRecentMinidump:439  No minidump found via crashpad_database_util
(INFO) 2017-04-26 00:35:02,837 desktop_browser_backend._GetMostRecentMinidump:444  Found minidump via globbing in minidump dir
(INFO) 2017-04-26 00:35:02,837 desktop_browser_backend.GetStackTrace:543  Minidump found: /b/s/w/itZwFQL8/tmp0riqc3/chromium-gpu-process-minidump-ff19bf1fa147f897.dmp
(INFO) 2017-04-26 00:35:02,837 cloud_storage.Insert:377  Uploading /b/s/w/itZwFQL8/tmp0riqc3/chromium-gpu-process-minidump-ff19bf1fa147f897.dmp to gs://chrome-telemetry-output/minidump-2017-04-26_00-35-02-418589.dmp
(INFO) 2017-04-26 00:35:03,618 desktop_browser_backend.GenerateBreakpadSymbols:79  Dumping breakpad symbols.
(WARNING) 2017-04-26 00:35:11,379 desktop_browser_backend._GetAllCrashpadMinidumps:366  No path to crashpad_database_util found
(INFO) 2017-04-26 00:35:11,379 desktop_browser_backend._GetMostRecentMinidump:439  No minidump found via crashpad_database_util
(INFO) 2017-04-26 00:35:11,379 desktop_browser_backend._GetMostRecentMinidump:444  Found minidump via globbing in minidump dir
(WARNING) 2017-04-26 00:35:11,380 screenshot.TryCaptureScreenShot:49  Exception when trying to capture screenshot: TimeoutException('',)
Traceback (most recent call last):
  File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/story_runner.py", line 88, in _RunStoryAndProcessErrorIfNeeded
    state.RunStory(results)
  File "/b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 52, in traced_function
    return func(*args, **kwargs)
  File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/page/shared_page_state.py", line 302, in RunStory
    self._current_page, self._current_tab, results)
  File "/b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/s/w/ir/tools/perf/measurements/smoothness.py", line 63, in ValidateAndMeasurePage
    self._tbm.Measure(tab.browser.platform, results)
  File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/web_perf/timeline_based_measurement.py", line 290, in Measure
    trace_result = platform.tracing_controller.StopTracing()
  File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/core/tracing_controller.py", line 47, in StopTracing
    return self._tracing_controller_backend.StopTracing()
  File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 140, in StopTracing
    '\n'.join(raised_exception_messages))
Error: Exceptions raised when trying to stop tracing:
Traceback (most recent call last):
  File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_controller_backend.py", line 118, in StopTracing
    agent.StopAgentTracing()
  File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 231, in StopAgentTracing
    '\n'.join(raised_exception_messages))
ChromeTracingStoppedError: Exceptions raised when trying to stop Chrome devtool tracing:
Error when trying to stop Chrome tracing on devtools at port 41530:
Traceback (most recent call last):
  File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/platform/tracing_agent/chrome_tracing_agent.py", line 214, in StopAgentTracing
    client.StopChromeTracing()
  File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/devtools_client_backend.py", line 359, in StopChromeTracing
    backend_id=backend.id)
  File "/b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 40, in inner
    inspector_backend._ConvertExceptionFromInspectorWebsocket(e)
  File "/b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
    return func(*args, **kwargs)
  File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 37, in inner
    return func(inspector_backend, *args, **kwargs)
  File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py", line 244, in EvaluateJavaScript
    return self._runtime.Evaluate(expression, context_id, timeout)
  File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_runtime.py", line 45, in Evaluate
    res = self._inspector_websocket.SyncRequest(request, timeout)
  File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 110, in SyncRequest
    res = self._Receive(timeout)
  File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_websocket.py", line 149, in _Receive
    data = self._socket.recv()
  File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 293, in recv
    opcode, data = self.recv_data()
  File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 310, in recv_data
    opcode, frame = self.recv_data_frame(control_frame)
  File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 323, in recv_data_frame
    frame = self.recv_frame()
  File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 357, in recv_frame
    return self.frame_buffer.recv_frame()
  File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 336, in recv_frame
    self.recv_header()
  File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 286, in recv_header
    header = self.recv_strict(2)
  File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_abnf.py", line 371, in recv_strict
    bytes_ = self.recv(min(16384, shortage))
  File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_core.py", line 427, in _recv
    return recv(self.sock, bufsize)
  File "/b/s/w/ir/third_party/catapult/telemetry/third_party/websocket-client/websocket/_socket.py", line 83, in recv
    raise WebSocketTimeoutException(message)
TimeoutException: 
********************************************************************************
(/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:477 _ConvertExceptionFromInspectorWebsocket) The app is probably crashed:

Found Minidump: True
Stack Trace:
********************************************************************************
	Operating system: Linux
	                  0.0.0 Linux 3.13.0-101-generic #148-Ubuntu SMP Thu Oct 20 22:08:32 UTC 2016 x86_64
	CPU: amd64
	     family 6 model 60 stepping 3
	     1 CPU
	
	GPU: UNKNOWN
	
	Crash reason:  SIGILL
	Crash address: 0x7ff02d7bec08
	Process uptime: not available
	
	Thread 0 (crashed)
	 0  libEGL.so!egl::MakeCurrent(void*, void*, void*, void*) [libEGL.cpp : 819 + 0x0]
	    rax = 0x00001b9b68176270   rdx = 0x00007ff02d7c4d60
	    rcx = 0x00007ff02d5e7320   rbx = 0x00001b9b68178cc0
	    rsi = 0x0000000000000001   rdi = 0x0000000000000000
	    rbp = 0x00001b9b68178c60   rsp = 0x00007ffe3f2b8160
	     r8 = 0x00001b9b681764e0    r9 = 0x0000000000000000
	    r10 = 0x00007ff026e8fe70   r11 = 0x00001b9b681cd01c
	    r12 = 0x0000000000000001   r13 = 0x00007ff02d7c2028
	    r14 = 0x00001b9b680ff000   r15 = 0x00001b9b68178cc0
	    rip = 0x00007ff02d7bec08
	    Found by: given as instruction pointer in context
	 1  chrome!gl::GLContextEGL::MakeCurrent(gl::GLSurface*) [gl_context_egl.cc : 192 + 0xf]
	    rbx = 0x00001b9b68178cc0   rbp = 0x00001b9b68178c60
	    rsp = 0x00007ffe3f2b8190   r12 = 0x00001b9b6813ff00
	    r13 = 0x00007ff0304a6d20   r14 = 0x00001b9b680fa6e0
	    r15 = 0x0000000000000001   rip = 0x00007ff0304a8e2e
	    Found by: call frame info
	 2  chrome!gpu::CollectGraphicsInfoGL(gpu::GPUInfo*) [gpu_info_collector.cc : 55 + 0x24]
	    rbx = 0x00007ff0348f66e0   rbp = 0x00007ffe3f2b8ae8
	    rsp = 0x00007ffe3f2b8390   r12 = 0x0000012449249249
	    r13 = 0x00000005fef61f9c   r14 = 0x00001b9b6813ff00
	    r15 = 0x00007ffe3f2b8ae8   rip = 0x00007ff02e4e5443
	    Found by: call frame info
	 3  chrome!gpu::CollectContextGraphicsInfo(gpu::GPUInfo*) [gpu_info_collector_linux.cc : 20 + 0x8]
	    rbx = 0x00007ff0348f66b0   rbp = 0x00007ffe3f2b8ae8
	    rsp = 0x00007ffe3f2b8620   r12 = 0x00000005fef61952
	    r13 = 0x00000005fef61f9c   r14 = 0x00007ffe3f2b8ae8
	    r15 = 0x0000000000000000   rip = 0x00007ff02e4f46dd
	    Found by: call frame info
	 4  chrome!<name omitted> [gpu_init.cc : 91 + 0x8]
	    rbx = 0x00007ffe3f2b8ad8   rbp = 0x00007ffe3f2b8ae8
	    rsp = 0x00007ffe3f2b86b0   r12 = 0x00000005fef61952
	    r13 = 0x00000005fef61f9c   r14 = 0x00001b9b680f9f00
	    r15 = 0x0000000000000000   rip = 0x00007ff030531586
	    Found by: call frame info
	 5  chrome!content::GpuMain(content::MainFunctionParams const&) [gpu_main.cc : 267 + 0xb]
	    rbx = 0x00007ffe3f2b8ad8   rbp = 0x00001b9b680f9f00
	    rsp = 0x00007ffe3f2b8900   r12 = 0x00007ff02e3780c4
	    r13 = 0x00001b9b68110a80   r14 = 0x00001b9b680f9f00
	    r15 = 0x00001b9b680fcea0   rip = 0x00007ff03226096a
	    Found by: call frame info
	 6  chrome!content::ContentMainRunnerImpl::Run() [content_main_runner.cc : 740 + 0x8]
	    rbx = 0x00001b9b68111e70   rbp = 0x00007ffe3f2b8ca0
	    rsp = 0x00007ffe3f2b8c80   r12 = 0x00007ff02e3780c4
	    r13 = 0x00007ffe3f2b9050   r14 = 0x00001b9b680f9f00
	    r15 = 0x00001b9b680fcea0   rip = 0x00007ff02f779675
	    Found by: call frame info
	 7  chrome!service_manager::Main(service_manager::MainParams const&) [main.cc : 179 + 0x8]
	    rbx = 0x00000000ffffffff   rbp = 0x0000000000000000
	    rsp = 0x00007ffe3f2b8cf0   r12 = 0x00007ff02e3780c4
	    r13 = 0x00007ffe3f2b9050   r14 = 0x00007ffe3f2b8e90
	    r15 = 0x00007ffe3f2b8e78   rip = 0x00007ff030a66330
	    Found by: call frame info
	 8  chrome!content::ContentMain(content::ContentMainParams const&) [content_main.cc : 19 + 0x8]
	    rbx = 0x00007ffe3f2b8f30   rbp = 0x0000000000000014
	    rsp = 0x00007ffe3f2b8e70   r12 = 0x00007ff02e3780c4
	    r13 = 0x00007ffe3f2b9050   r14 = 0x00007ffe3f2b8e90
	    r15 = 0x00007ffe3f2b8e78   rip = 0x00007ff02f77a3b2
	    Found by: call frame info
	 9  chrome!ChromeMain [chrome_main.cc : 123 + 0x5]
	    rbx = 0x00007ffe3f2b9058   rbp = 0x0000000000000014
	    rsp = 0x00007ffe3f2b8ef0   r12 = 0x00007ff02e3780c4
	    r13 = 0x00007ffe3f2b9050   r14 = 0x00007ffe3f2b8ef8
	    r15 = 0x0000000000000000   rip = 0x00007ff02e3782b1
	    Found by: call frame info
	10  libc-2.19.so + 0x21ec5
	    rbx = 0x0000000000000000   rbp = 0x0000000000000000
	    rsp = 0x00007ffe3f2b8f80   r12 = 0x00007ff02e3780c4
	    r13 = 0x00007ffe3f2b9050   r14 = 0x0000000000000000
	    r15 = 0x0000000000000000   rip = 0x00007ff026d70ec5
	    Found by: call frame info
	11  chrome!SyscallAsm + 0x30
	    rsp = 0x00007ffe3f2b8fa0   rip = 0x00007ff02e378220
	    Found by: stack scanning
	12  chrome!__cxx_global_array_dtor [os2.cc : 103 + 0x14]
	    rsp = 0x00007ffe3f2b8fb8   rip = 0x00007ff02e3780c4
	    Found by: stack scanning
	
	Thread 1
	 0  libpthread-2.19.so + 0xc7ce
	    rax = 0xfffffffffffffdfc   rdx = 0x0000000000000001
	    rcx = 0xffffffffffffffff   rbx = 0x00000000000064a2
	    rsi = 0x0000000000000089   rdi = 0x00007ff020ae4a7c
	    rbp = 0x7fffffffffffffff   rsp = 0x00007ff020ae4990
	     r8 = 0x00007ff020ae4a50    r9 = 0x00000000ffffffff
	    r10 = 0x00007ff020ae4a00   r11 = 0x0000000000000202
	    r12 = 0x0000000000000001   r13 = 0x00007ff020ae4a00
	    r14 = 0xffffffffffffff92   r15 = 0x000000001683fd00
	    rip = 0x00007ff02d3da7ce
	    Found by: given as instruction pointer in context
	 1  chrome!<name omitted> [condition_variable_posix.cc : 117 + 0x8]
	    rbp = 0x7fffffffffffffff   rsp = 0x00007ff020ae49f0
	    rip = 0x00007ff02fafadb2
	    Found by: stack scanning
	 2  chrome!<name omitted> [waitable_event_posix.cc : 223 + 0xd]
	    rbx = 0x00007ff020ae4a30   rbp = 0x7fffffffffffffff
	    rsp = 0x00007ff020ae4a30   r14 = 0x00000005ff8eb010
	    r15 = 0x00007ff020ae4a38   rip = 0x00007ff02fafb23a
	    Found by: call frame info
	 3  chrome!base::MessagePumpDefault::Run(base::MessagePump::Delegate*) [message_pump_default.cc : 59 + 0xb]
	    rbx = 0x0000000000000000   rbp = 0x00007ff0345649a0
	    rsp = 0x00007ff020ae4b00   r12 = 0x00001b9b6813c2b0
	    r13 = 0x00001b9b681108c0   r14 = 0x00001b9b6813c2a0
	    r15 = 0x00001b9b6813c2b8   rip = 0x00007ff02faceb38
	    Found by: call frame info
	 4  chrome!<name omitted> [run_loop.cc : 37 + 0x5]
	    rbx = 0x00007ff020ae4bd8   rbp = 0x00007ff034750860
	    rsp = 0x00007ff020ae4b40   r12 = 0x0000000000000000
	    r13 = 0x00001b9b68107c70   r14 = 0x00007ff020ae4b40
	    r15 = 0x00007ff020ae4bd8   rip = 0x00007ff02faeb54e
	    Found by: call frame info
	 5  chrome!<name omitted> [thread.cc : 333 + 0x6]
	    rbx = 0x00001b9b68107c60   rbp = 0x00007ff034750860
	    rsp = 0x00007ff020ae4bc0   r12 = 0x0000000000000000
	    r13 = 0x00001b9b68107c70   r14 = 0x00001b9b681108c0
	    r15 = 0x00007ff020ae4bd8   rip = 0x00007ff02fb0d49f
	    Found by: call frame info
	 6  chrome!base::(anonymous namespace)::ThreadFunc(void*) [platform_thread_posix.cc : 71 + 0x29]
	    rbx = 0x00007ff020ae5700   rbp = 0x00001b9b68113b60
	    rsp = 0x00007ff020ae4c40   r12 = 0x0000000000000000
	    r13 = 0x0000000000000000   r14 = 0x00001b9b68107c60
	    r15 = 0x00007ff020ae5700   rip = 0x00007ff02fb09214
	    Found by: call frame info
	 7  libpthread-2.19.so + 0x8182
	    rbx = 0x00007ff020ae5700   rbp = 0x0000000000000000
	    rsp = 0x00007ff020ae4c60   r12 = 0x0000000000000000
	    r13 = 0x0000000000000000   r14 = 0x00007ff020ae59c0
	    r15 = 0x00007ff020ae5700   rip = 0x00007ff02d3d6182
	    Found by: call frame info
	 8  libc-2.19.so + 0xfa47d
	    rsp = 0x00007ff020ae4d00   rip = 0x00007ff026e4947d
	    Found by: stack scanning
	
	Loaded modules:
	0x7ff02000d000 - 0x7ff02027ffff  libGLESv2.so  ???
	0x7ff020ae6000 - 0x7ff02148ffff  icudtl.dat  ???
	0x7ff021490000 - 0x7ff02194afff  locale-archive  ???
	0x7ff02194b000 - 0x7ff02194efff  libgpg-error.so.0.10.0  ???
	0x7ff021b50000 - 0x7ff021b51fff  libkeyutils.so.1.4  ???
	0x7ff021d54000 - 0x7ff021d6dfff  libgraphite2.so.3.0.1  ???
	0x7ff021f70000 - 0x7ff021f97fff  libatspi.so.0.0.1  ???
	0x7ff02219c000 - 0x7ff0221d6fff  libp11-kit.so.0.0.0  ???
	0x7ff0223de000 - 0x7ff0223effff  libtasn1.so.6.2.0  ???
	0x7ff0225f2000 - 0x7ff02266dfff  libgcrypt.so.11.8.2  ???
	0x7ff022872000 - 0x7ff02287bfff  libkrb5support.so.0.1  ???
	0x7ff022a7d000 - 0x7ff022a7ffff  libcom_err.so.2.1  ???
	0x7ff022c81000 - 0x7ff022cacfff  libk5crypto.so.3.1  ???
	0x7ff022eb0000 - 0x7ff022f6bfff  libkrb5.so.3.3  ???
	0x7ff02317b000 - 0x7ff023180fff  libdatrie.so.1.3.1  ???
	0x7ff023382000 - 0x7ff0233d4fff  libharfbuzz.so.0.927.0  ???
	0x7ff0235d7000 - 0x7ff0235dafff  libwayland-cursor.so.0.0.0  ???
	0x7ff0237df000 - 0x7ff023816fff  libxkbcommon.so.0.0.0  ???
	0x7ff023a19000 - 0x7ff023a23fff  libwayland-client.so.0.2.0  ???
	0x7ff023c26000 - 0x7ff023c27fff  libXinerama.so.1.0.0  ???
	0x7ff023e29000 - 0x7ff023e50fff  libatk-bridge-2.0.so.0.0.0  ???
	0x7ff024054000 - 0x7ff02405afff  libcairo-gobject.so.2.11301.0  ???
	0x7ff02425d000 - 0x7ff024281fff  libdbus-glib-1.so.2.2.2  ???
	0x7ff024484000 - 0x7ff02449afff  libresolv-2.19.so  ???
	0x7ff02469f000 - 0x7ff0246befff  libselinux.so.1  ???
	0x7ff0248c2000 - 0x7ff0248d0fff  libavahi-client.so.3.2.9  ???
	0x7ff024ad3000 - 0x7ff024addfff  libavahi-common.so.3.5.3  ???
	0x7ff024cdf000 - 0x7ff024d94fff  libgnutls.so.26.22.6  ???
	0x7ff024f9d000 - 0x7ff024fe0fff  libgssapi_krb5.so.2.2  ???
	0x7ff0251e4000 - 0x7ff0251e6fff  libplds4.so  ???
	0x7ff0253e8000 - 0x7ff0253ebfff  libplc4.so  ???
	0x7ff0255ed000 - 0x7ff0255f1fff  libXdmcp.so.6.0.0  ???
	0x7ff0257f3000 - 0x7ff0257f4fff  libXau.so.6.0.0  ???
	0x7ff0259f7000 - 0x7ff025a33fff  libpcre.so.3.13.1  ???
	0x7ff025c35000 - 0x7ff025c3bfff  libffi.so.6.0.1  ???
	0x7ff025e3d000 - 0x7ff025e54fff  libz.so.1.2.8  ???
	0x7ff026056000 - 0x7ff02605dfff  libxcb-render.so.0.0.0  ???
	0x7ff02625f000 - 0x7ff026260fff  libxcb-shm.so.0.0.0  ???
	0x7ff026462000 - 0x7ff026486fff  libpng12.so.0.50.0  ???
	0x7ff026688000 - 0x7ff026728fff  libpixman-1.so.0.30.2  ???
	0x7ff026931000 - 0x7ff026938fff  libthai.so.0.2.0  ???
	0x7ff026b3a000 - 0x7ff026b4dfff  libpangoft2-1.0.so.0.3600.3  ???
	0x7ff026d4f000 - 0x7ff026f09fff  libc-2.19.so  ???  (WARNING: No symbols, libc-2.19.so, 8B0582530369AA1C9B9996C11061CD160)
	0x7ff027114000 - 0x7ff027129fff  libgcc_s.so.1  ???
	0x7ff02732a000 - 0x7ff02740ffff  libstdc++.so.6.0.19  ???
	0x7ff02762e000 - 0x7ff02764dfff  libgdk_pixbuf-2.0.so.0.3000.7  ???
	0x7ff02784f000 - 0x7ff0278eefff  libgdk-3.so.0.1000.8  ???
	0x7ff027af6000 - 0x7ff027ff5fff  libgtk-3.so.0.1000.8  ???
	0x7ff028207000 - 0x7ff028225fff  libatk-1.0.so.0.21009.1  ???
	0x7ff028429000 - 0x7ff02852dfff  libm-2.19.so  ???
	0x7ff02872f000 - 0x7ff028817fff  libasound.so.2.0.0  ???
	0x7ff028a1f000 - 0x7ff028a27fff  libXrandr.so.2.2.0  ???
	0x7ff028c29000 - 0x7ff028c2afff  libXss.so.1.0.0  ???
	0x7ff028e2d000 - 0x7ff028ec9fff  libfreetype.so.6.11.1  ???
	0x7ff0290d0000 - 0x7ff0290fdfff  libgconf-2.so.4.1.5  ???
	0x7ff0292ff000 - 0x7ff02946afff  libgio-2.0.so.0.4002.0  ???
	0x7ff029672000 - 0x7ff0296b5fff  libdbus-1.so.3.7.6  ???
	0x7ff0298b7000 - 0x7ff0298f0fff  libfontconfig.so.1.8.0  ???
	0x7ff029af3000 - 0x7ff029b19fff  libexpat.so.1.6.0  ???
	0x7ff029d1d000 - 0x7ff029d88fff  libcups.so.2  ???
	0x7ff029f91000 - 0x7ff029fc9fff  libnspr4.so  ???
	0x7ff02a1ce000 - 0x7ff02a1f5fff  libsmime3.so  ???
	0x7ff02a3fa000 - 0x7ff02a41efff  libnssutil3.so  ???
	0x7ff02a626000 - 0x7ff02a75afff  libnss3.so  ???
	0x7ff02a962000 - 0x7ff02a964fff  libgmodule-2.0.so.0.4002.0  ???
	0x7ff02ab66000 - 0x7ff02ab6afff  libXtst.so.6.1.0  ???
	0x7ff02ad6c000 - 0x7ff02ad74fff  libXrender.so.1.3.0  ???
	0x7ff02af76000 - 0x7ff02af84fff  libXi.so.6.1.0  ???
	0x7ff02b186000 - 0x7ff02b18afff  libXfixes.so.3.1.0  ???
	0x7ff02b38c000 - 0x7ff02b39cfff  libXext.so.6.4.0  ???
	0x7ff02b59e000 - 0x7ff02b59ffff  libXdamage.so.1.1.0  ???
	0x7ff02b7a1000 - 0x7ff02b7a9fff  libXcursor.so.1.0.2  ???
	0x7ff02b9ab000 - 0x7ff02b9acfff  libXcomposite.so.1.0.0  ???
	0x7ff02bbae000 - 0x7ff02bbcafff  libxcb.so.1.1.0  ???
	0x7ff02bdcd000 - 0x7ff02bdcdfff  libX11-xcb.so.1.0.0  ???
	0x7ff02bfcf000 - 0x7ff02c0fefff  libX11.so.6.3.0  ???
	0x7ff02c304000 - 0x7ff02c409fff  libglib-2.0.so.0.4002.0  ???
	0x7ff02c60c000 - 0x7ff02c65afff  libgobject-2.0.so.0.4002.0  ???
	0x7ff02c85d000 - 0x7ff02c962fff  libcairo.so.2.11301.0  ???
	0x7ff02cb68000 - 0x7ff02cbb1fff  libpango-1.0.so.0.3600.3  ???
	0x7ff02cdb5000 - 0x7ff02cdbffff  libpangocairo-1.0.so.0.3600.3  ???
	0x7ff02cfc2000 - 0x7ff02cfc8fff  librt-2.19.so  ???
	0x7ff02d1ca000 - 0x7ff02d1ccfff  libdl-2.19.so  ???
	0x7ff02d3ce000 - 0x7ff02d3e6fff  libpthread-2.19.so  ???  (WARNING: No symbols, libpthread-2.19.so, 860606AF75813607E0524DF17D5A86CB0)
	0x7ff02d5ec000 - 0x7ff02d60efff  ld-2.19.so  ???
	0x7ff02d7b9000 - 0x7ff02d7c0fff  libEGL.so  ???
	0x7ff02d7ec000 - 0x7ff02d80bfff  .com.google.Chrome.m5vZG0 (deleted)  ???
	0x7ff02d811000 - 0x7ff03433cfff  chrome  ???  (main)
	0x7ffe3f37e000 - 0x7ffe3f37ffff  linux-gate.so  ???
********************************************************************************
Standard output:
********************************************************************************
	[22001:22001:0426/003054.348627:ERROR:sandbox_linux.cc(344)] InitializeSandbox() called with multiple threads in process gpu-process.
********************************************************************************
System log:
(Not implemented)
********************************************************************************
(/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:482 _ConvertExceptionFromInspectorWebsocket) Original exception:
timed out
********************************************************************************
(/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:503 _AddDebuggingInformation) Received a socket error in the browser connection and the tab still exists. The operation probably timed out.
********************************************************************************
(/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:504 _AddDebuggingInformation) Debugger url: ws://127.0.0.1:41530/devtools/page/195bf043-c693-4c78-b320-3a551bfb4039



(WARNING) 2017-04-26 00:36:11,442 desktop_browser_backend._GetAllCrashpadMinidumps:366  No path to crashpad_database_util found
(INFO) 2017-04-26 00:36:11,442 desktop_browser_backend._GetMostRecentMinidump:439  No minidump found via crashpad_database_util
(INFO) 2017-04-26 00:36:11,442 desktop_browser_backend._GetMostRecentMinidump:444  Found minidump via globbing in minidump dir
(WARNING) 2017-04-26 00:36:11,443 desktop_browser_backend._GetMostRecentMinidump:449  Crash dump is older than 5 minutes. May not be correct.
(INFO) 2017-04-26 00:36:11,443 desktop_browser_backend.GetStackTrace:543  Minidump found: /b/s/w/itZwFQL8/tmp0riqc3/chromium-gpu-process-minidump-ff19bf1fa147f897.dmp
(INFO) 2017-04-26 00:36:11,443 cloud_storage.Insert:377  Uploading /b/s/w/itZwFQL8/tmp0riqc3/chromium-gpu-process-minidump-ff19bf1fa147f897.dmp to gs://chrome-telemetry-output/minidump-2017-04-26_00-36-11-301082.dmp
(INFO) 2017-04-26 00:36:13,263 desktop_browser_backend.GenerateBreakpadSymbols:79  Dumping breakpad symbols.
(WARNING) 2017-04-26 00:36:20,768 desktop_browser_backend._GetAllCrashpadMinidumps:366  No path to crashpad_database_util found
(INFO) 2017-04-26 00:36:20,768 desktop_browser_backend._GetMostRecentMinidump:439  No minidump found via crashpad_database_util
(INFO) 2017-04-26 00:36:20,769 desktop_browser_backend._GetMostRecentMinidump:444  Found minidump via globbing in minidump dir
(WARNING) 2017-04-26 00:36:20,769 desktop_browser_backend._GetMostRecentMinidump:449  Crash dump is older than 5 minutes. May not be correct.
(WARNING) 2017-04-26 00:38:20,876 desktop_browser_backend._GetAllCrashpadMinidumps:366  No path to crashpad_database_util found
(INFO) 2017-04-26 00:38:20,876 desktop_browser_backend._GetMostRecentMinidump:439  No minidump found via crashpad_database_util
(INFO) 2017-04-26 00:38:20,876 desktop_browser_backend._GetMostRecentMinidump:444  Found minidump via globbing in minidump dir
(WARNING) 2017-04-26 00:38:20,876 desktop_browser_backend._GetMostRecentMinidump:449  Crash dump is older than 5 minutes. May not be correct.
(INFO) 2017-04-26 00:38:20,876 desktop_browser_backend.GetStackTrace:543  Minidump found: /b/s/w/itZwFQL8/tmp0riqc3/chromium-gpu-process-minidump-ff19bf1fa147f897.dmp
(INFO) 2017-04-26 00:38:20,876 cloud_storage.Insert:377  Uploading /b/s/w/itZwFQL8/tmp0riqc3/chromium-gpu-process-minidump-ff19bf1fa147f897.dmp to gs://chrome-telemetry-output/minidump-2017-04-26_00-38-20-91847.dmp
(INFO) 2017-04-26 00:38:21,848 desktop_browser_backend.GenerateBreakpadSymbols:79  Dumping breakpad symbols.
(WARNING) 2017-04-26 00:38:29,515 desktop_browser_backend._GetAllCrashpadMinidumps:366  No path to crashpad_database_util found
(INFO) 2017-04-26 00:38:29,515 desktop_browser_backend._GetMostRecentMinidump:439  No minidump found via crashpad_database_util
(INFO) 2017-04-26 00:38:29,515 desktop_browser_backend._GetMostRecentMinidump:444  Found minidump via globbing in minidump dir
(WARNING) 2017-04-26 00:38:29,515 desktop_browser_backend._GetMostRecentMinidump:449  Crash dump is older than 5 minutes. May not be correct.
(INFO) 2017-04-26 00:38:29,516 desktop_browser_backend.HasBrowserFinishedLaunching:238  Discovered ephemeral port 41530
(INFO) 2017-04-26 00:38:29,517 exception_formatter.PrintFormattedException:19  Try printing formatted exception: None None None

Exception raised when cleaning story run: 

Traceback (most recent call last):
  _RunStoryAndProcessErrorIfNeeded at /b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/story_runner.py:107
    state.DidRunStory(results)
  traced_function at /b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
    return func(*args, **kwargs)
  DidRunStory at /b/s/w/ir/third_party/catapult/telemetry/telemetry/page/shared_page_state.py:154
    self._current_tab.Close()
  traced_function at /b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py:52
    return func(*args, **kwargs)
  Close at /b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/tab.py:100
    self._tab_list_backend.CloseTab(self.id)
  CloseTab at /b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome/tab_list_backend.py:58
    self.New(timeout)
  New at /b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome/tab_list_backend.py:43
    return self.GetBackendFromContextId(context_id)
  GetBackendFromContextId at /b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend_list.py:78
    raise e
TimeoutException: 
********************************************************************************
(/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:477 _ConvertExceptionFromInspectorWebsocket) The app is probably crashed:

Found Minidump: True
Stack Trace:
********************************************************************************
	Operating system: Linux
	                  0.0.0 Linux 3.13.0-101-generic #148-Ubuntu SMP Thu Oct 20 22:08:32 UTC 2016 x86_64
	CPU: amd64
	     family 6 model 60 stepping 3
	     1 CPU
	
	GPU: UNKNOWN
	
	Crash reason:  SIGILL
	Crash address: 0x7ff02d7bec08
	Process uptime: not available
	
	Thread 0 (crashed)
	 0  libEGL.so!egl::MakeCurrent(void*, void*, void*, void*) [libEGL.cpp : 819 + 0x0]
	    rax = 0x00001b9b68176270   rdx = 0x00007ff02d7c4d60
	    rcx = 0x00007ff02d5e7320   rbx = 0x00001b9b68178cc0
	    rsi = 0x0000000000000001   rdi = 0x0000000000000000
	    rbp = 0x00001b9b68178c60   rsp = 0x00007ffe3f2b8160
	     r8 = 0x00001b9b681764e0    r9 = 0x0000000000000000
	    r10 = 0x00007ff026e8fe70   r11 = 0x00001b9b681cd01c
	    r12 = 0x0000000000000001   r13 = 0x00007ff02d7c2028
	    r14 = 0x00001b9b680ff000   r15 = 0x00001b9b68178cc0
	    rip = 0x00007ff02d7bec08
	    Found by: given as instruction pointer in context
	 1  chrome!gl::GLContextEGL::MakeCurrent(gl::GLSurface*) [gl_context_egl.cc : 192 + 0xf]
	    rbx = 0x00001b9b68178cc0   rbp = 0x00001b9b68178c60
	    rsp = 0x00007ffe3f2b8190   r12 = 0x00001b9b6813ff00
	    r13 = 0x00007ff0304a6d20   r14 = 0x00001b9b680fa6e0
	    r15 = 0x0000000000000001   rip = 0x00007ff0304a8e2e
	    Found by: call frame info
	 2  chrome!gpu::CollectGraphicsInfoGL(gpu::GPUInfo*) [gpu_info_collector.cc : 55 + 0x24]
	    rbx = 0x00007ff0348f66e0   rbp = 0x00007ffe3f2b8ae8
	    rsp = 0x00007ffe3f2b8390   r12 = 0x0000012449249249
	    r13 = 0x00000005fef61f9c   r14 = 0x00001b9b6813ff00
	    r15 = 0x00007ffe3f2b8ae8   rip = 0x00007ff02e4e5443
	    Found by: call frame info
	 3  chrome!gpu::CollectContextGraphicsInfo(gpu::GPUInfo*) [gpu_info_collector_linux.cc : 20 + 0x8]
	    rbx = 0x00007ff0348f66b0   rbp = 0x00007ffe3f2b8ae8
	    rsp = 0x00007ffe3f2b8620   r12 = 0x00000005fef61952
	    r13 = 0x00000005fef61f9c   r14 = 0x00007ffe3f2b8ae8
	    r15 = 0x0000000000000000   rip = 0x00007ff02e4f46dd
	    Found by: call frame info
	 4  chrome!<name omitted> [gpu_init.cc : 91 + 0x8]
	    rbx = 0x00007ffe3f2b8ad8   rbp = 0x00007ffe3f2b8ae8
	    rsp = 0x00007ffe3f2b86b0   r12 = 0x00000005fef61952
	    r13 = 0x00000005fef61f9c   r14 = 0x00001b9b680f9f00
	    r15 = 0x0000000000000000   rip = 0x00007ff030531586
	    Found by: call frame info
	 5  chrome!content::GpuMain(content::MainFunctionParams const&) [gpu_main.cc : 267 + 0xb]
	    rbx = 0x00007ffe3f2b8ad8   rbp = 0x00001b9b680f9f00
	    rsp = 0x00007ffe3f2b8900   r12 = 0x00007ff02e3780c4
	    r13 = 0x00001b9b68110a80   r14 = 0x00001b9b680f9f00
	    r15 = 0x00001b9b680fcea0   rip = 0x00007ff03226096a
	    Found by: call frame info
	 6  chrome!content::ContentMainRunnerImpl::Run() [content_main_runner.cc : 740 + 0x8]
	    rbx = 0x00001b9b68111e70   rbp = 0x00007ffe3f2b8ca0
	    rsp = 0x00007ffe3f2b8c80   r12 = 0x00007ff02e3780c4
	    r13 = 0x00007ffe3f2b9050   r14 = 0x00001b9b680f9f00
	    r15 = 0x00001b9b680fcea0   rip = 0x00007ff02f779675
	    Found by: call frame info
	 7  chrome!service_manager::Main(service_manager::MainParams const&) [main.cc : 179 + 0x8]
	    rbx = 0x00000000ffffffff   rbp = 0x0000000000000000
	    rsp = 0x00007ffe3f2b8cf0   r12 = 0x00007ff02e3780c4
	    r13 = 0x00007ffe3f2b9050   r14 = 0x00007ffe3f2b8e90
	    r15 = 0x00007ffe3f2b8e78   rip = 0x00007ff030a66330
	    Found by: call frame info
	 8  chrome!content::ContentMain(content::ContentMainParams const&) [content_main.cc : 19 + 0x8]
	    rbx = 0x00007ffe3f2b8f30   rbp = 0x0000000000000014
	    rsp = 0x00007ffe3f2b8e70   r12 = 0x00007ff02e3780c4
	    r13 = 0x00007ffe3f2b9050   r14 = 0x00007ffe3f2b8e90
	    r15 = 0x00007ffe3f2b8e78   rip = 0x00007ff02f77a3b2
	    Found by: call frame info
	 9  chrome!ChromeMain [chrome_main.cc : 123 + 0x5]
	    rbx = 0x00007ffe3f2b9058   rbp = 0x0000000000000014
	    rsp = 0x00007ffe3f2b8ef0   r12 = 0x00007ff02e3780c4
	    r13 = 0x00007ffe3f2b9050   r14 = 0x00007ffe3f2b8ef8
	    r15 = 0x0000000000000000   rip = 0x00007ff02e3782b1
	    Found by: call frame info
	10  libc-2.19.so + 0x21ec5
	    rbx = 0x0000000000000000   rbp = 0x0000000000000000
	    rsp = 0x00007ffe3f2b8f80   r12 = 0x00007ff02e3780c4
	    r13 = 0x00007ffe3f2b9050   r14 = 0x0000000000000000
	    r15 = 0x0000000000000000   rip = 0x00007ff026d70ec5
	    Found by: call frame info
	11  chrome!SyscallAsm + 0x30
	    rsp = 0x00007ffe3f2b8fa0   rip = 0x00007ff02e378220
	    Found by: stack scanning
	12  chrome!__cxx_global_array_dtor [os2.cc : 103 + 0x14]
	    rsp = 0x00007ffe3f2b8fb8   rip = 0x00007ff02e3780c4
	    Found by: stack scanning
	
	Thread 1
	 0  libpthread-2.19.so + 0xc7ce
	    rax = 0xfffffffffffffdfc   rdx = 0x0000000000000001
	    rcx = 0xffffffffffffffff   rbx = 0x00000000000064a2
	    rsi = 0x0000000000000089   rdi = 0x00007ff020ae4a7c
	    rbp = 0x7fffffffffffffff   rsp = 0x00007ff020ae4990
	     r8 = 0x00007ff020ae4a50    r9 = 0x00000000ffffffff
	    r10 = 0x00007ff020ae4a00   r11 = 0x0000000000000202
	    r12 = 0x0000000000000001   r13 = 0x00007ff020ae4a00
	    r14 = 0xffffffffffffff92   r15 = 0x000000001683fd00
	    rip = 0x00007ff02d3da7ce
	    Found by: given as instruction pointer in context
	 1  chrome!<name omitted> [condition_variable_posix.cc : 117 + 0x8]
	    rbp = 0x7fffffffffffffff   rsp = 0x00007ff020ae49f0
	    rip = 0x00007ff02fafadb2
	    Found by: stack scanning
	 2  chrome!<name omitted> [waitable_event_posix.cc : 223 + 0xd]
	    rbx = 0x00007ff020ae4a30   rbp = 0x7fffffffffffffff
	    rsp = 0x00007ff020ae4a30   r14 = 0x00000005ff8eb010
	    r15 = 0x00007ff020ae4a38   rip = 0x00007ff02fafb23a
	    Found by: call frame info
	 3  chrome!base::MessagePumpDefault::Run(base::MessagePump::Delegate*) [message_pump_default.cc : 59 + 0xb]
	    rbx = 0x0000000000000000   rbp = 0x00007ff0345649a0
	    rsp = 0x00007ff020ae4b00   r12 = 0x00001b9b6813c2b0
	    r13 = 0x00001b9b681108c0   r14 = 0x00001b9b6813c2a0
	    r15 = 0x00001b9b6813c2b8   rip = 0x00007ff02faceb38
	    Found by: call frame info
	 4  chrome!<name omitted> [run_loop.cc : 37 + 0x5]
	    rbx = 0x00007ff020ae4bd8   rbp = 0x00007ff034750860
	    rsp = 0x00007ff020ae4b40   r12 = 0x0000000000000000
	    r13 = 0x00001b9b68107c70   r14 = 0x00007ff020ae4b40
	    r15 = 0x00007ff020ae4bd8   rip = 0x00007ff02faeb54e
	    Found by: call frame info
	 5  chrome!<name omitted> [thread.cc : 333 + 0x6]
	    rbx = 0x00001b9b68107c60   rbp = 0x00007ff034750860
	    rsp = 0x00007ff020ae4bc0   r12 = 0x0000000000000000
	    r13 = 0x00001b9b68107c70   r14 = 0x00001b9b681108c0
	    r15 = 0x00007ff020ae4bd8   rip = 0x00007ff02fb0d49f
	    Found by: call frame info
	 6  chrome!base::(anonymous namespace)::ThreadFunc(void*) [platform_thread_posix.cc : 71 + 0x29]
	    rbx = 0x00007ff020ae5700   rbp = 0x00001b9b68113b60
	    rsp = 0x00007ff020ae4c40   r12 = 0x0000000000000000
	    r13 = 0x0000000000000000   r14 = 0x00001b9b68107c60
	    r15 = 0x00007ff020ae5700   rip = 0x00007ff02fb09214
	    Found by: call frame info
	 7  libpthread-2.19.so + 0x8182
	    rbx = 0x00007ff020ae5700   rbp = 0x0000000000000000
	    rsp = 0x00007ff020ae4c60   r12 = 0x0000000000000000
	    r13 = 0x0000000000000000   r14 = 0x00007ff020ae59c0
	    r15 = 0x00007ff020ae5700   rip = 0x00007ff02d3d6182
	    Found by: call frame info
	 8  libc-2.19.so + 0xfa47d
	    rsp = 0x00007ff020ae4d00   rip = 0x00007ff026e4947d
	    Found by: stack scanning
	
	Loaded modules:
	0x7ff02000d000 - 0x7ff02027ffff  libGLESv2.so  ???
	0x7ff020ae6000 - 0x7ff02148ffff  icudtl.dat  ???
	0x7ff021490000 - 0x7ff02194afff  locale-archive  ???
	0x7ff02194b000 - 0x7ff02194efff  libgpg-error.so.0.10.0  ???
	0x7ff021b50000 - 0x7ff021b51fff  libkeyutils.so.1.4  ???
	0x7ff021d54000 - 0x7ff021d6dfff  libgraphite2.so.3.0.1  ???
	0x7ff021f70000 - 0x7ff021f97fff  libatspi.so.0.0.1  ???
	0x7ff02219c000 - 0x7ff0221d6fff  libp11-kit.so.0.0.0  ???
	0x7ff0223de000 - 0x7ff0223effff  libtasn1.so.6.2.0  ???
	0x7ff0225f2000 - 0x7ff02266dfff  libgcrypt.so.11.8.2  ???
	0x7ff022872000 - 0x7ff02287bfff  libkrb5support.so.0.1  ???
	0x7ff022a7d000 - 0x7ff022a7ffff  libcom_err.so.2.1  ???
	0x7ff022c81000 - 0x7ff022cacfff  libk5crypto.so.3.1  ???
	0x7ff022eb0000 - 0x7ff022f6bfff  libkrb5.so.3.3  ???
	0x7ff02317b000 - 0x7ff023180fff  libdatrie.so.1.3.1  ???
	0x7ff023382000 - 0x7ff0233d4fff  libharfbuzz.so.0.927.0  ???
	0x7ff0235d7000 - 0x7ff0235dafff  libwayland-cursor.so.0.0.0  ???
	0x7ff0237df000 - 0x7ff023816fff  libxkbcommon.so.0.0.0  ???
	0x7ff023a19000 - 0x7ff023a23fff  libwayland-client.so.0.2.0  ???
	0x7ff023c26000 - 0x7ff023c27fff  libXinerama.so.1.0.0  ???
	0x7ff023e29000 - 0x7ff023e50fff  libatk-bridge-2.0.so.0.0.0  ???
	0x7ff024054000 - 0x7ff02405afff  libcairo-gobject.so.2.11301.0  ???
	0x7ff02425d000 - 0x7ff024281fff  libdbus-glib-1.so.2.2.2  ???
	0x7ff024484000 - 0x7ff02449afff  libresolv-2.19.so  ???
	0x7ff02469f000 - 0x7ff0246befff  libselinux.so.1  ???
	0x7ff0248c2000 - 0x7ff0248d0fff  libavahi-client.so.3.2.9  ???
	0x7ff024ad3000 - 0x7ff024addfff  libavahi-common.so.3.5.3  ???
	0x7ff024cdf000 - 0x7ff024d94fff  libgnutls.so.26.22.6  ???
	0x7ff024f9d000 - 0x7ff024fe0fff  libgssapi_krb5.so.2.2  ???
	0x7ff0251e4000 - 0x7ff0251e6fff  libplds4.so  ???
	0x7ff0253e8000 - 0x7ff0253ebfff  libplc4.so  ???
	0x7ff0255ed000 - 0x7ff0255f1fff  libXdmcp.so.6.0.0  ???
	0x7ff0257f3000 - 0x7ff0257f4fff  libXau.so.6.0.0  ???
	0x7ff0259f7000 - 0x7ff025a33fff  libpcre.so.3.13.1  ???
	0x7ff025c35000 - 0x7ff025c3bfff  libffi.so.6.0.1  ???
	0x7ff025e3d000 - 0x7ff025e54fff  libz.so.1.2.8  ???
	0x7ff026056000 - 0x7ff02605dfff  libxcb-render.so.0.0.0  ???
	0x7ff02625f000 - 0x7ff026260fff  libxcb-shm.so.0.0.0  ???
	0x7ff026462000 - 0x7ff026486fff  libpng12.so.0.50.0  ???
	0x7ff026688000 - 0x7ff026728fff  libpixman-1.so.0.30.2  ???
	0x7ff026931000 - 0x7ff026938fff  libthai.so.0.2.0  ???
	0x7ff026b3a000 - 0x7ff026b4dfff  libpangoft2-1.0.so.0.3600.3  ???
	0x7ff026d4f000 - 0x7ff026f09fff  libc-2.19.so  ???  (WARNING: No symbols, libc-2.19.so, 8B0582530369AA1C9B9996C11061CD160)
	0x7ff027114000 - 0x7ff027129fff  libgcc_s.so.1  ???
	0x7ff02732a000 - 0x7ff02740ffff  libstdc++.so.6.0.19  ???
	0x7ff02762e000 - 0x7ff02764dfff  libgdk_pixbuf-2.0.so.0.3000.7  ???
	0x7ff02784f000 - 0x7ff0278eefff  libgdk-3.so.0.1000.8  ???
	0x7ff027af6000 - 0x7ff027ff5fff  libgtk-3.so.0.1000.8  ???
	0x7ff028207000 - 0x7ff028225fff  libatk-1.0.so.0.21009.1  ???
	0x7ff028429000 - 0x7ff02852dfff  libm-2.19.so  ???
	0x7ff02872f000 - 0x7ff028817fff  libasound.so.2.0.0  ???
	0x7ff028a1f000 - 0x7ff028a27fff  libXrandr.so.2.2.0  ???
	0x7ff028c29000 - 0x7ff028c2afff  libXss.so.1.0.0  ???
	0x7ff028e2d000 - 0x7ff028ec9fff  libfreetype.so.6.11.1  ???
	0x7ff0290d0000 - 0x7ff0290fdfff  libgconf-2.so.4.1.5  ???
	0x7ff0292ff000 - 0x7ff02946afff  libgio-2.0.so.0.4002.0  ???
	0x7ff029672000 - 0x7ff0296b5fff  libdbus-1.so.3.7.6  ???
	0x7ff0298b7000 - 0x7ff0298f0fff  libfontconfig.so.1.8.0  ???
	0x7ff029af3000 - 0x7ff029b19fff  libexpat.so.1.6.0  ???
	0x7ff029d1d000 - 0x7ff029d88fff  libcups.so.2  ???
	0x7ff029f91000 - 0x7ff029fc9fff  libnspr4.so  ???
	0x7ff02a1ce000 - 0x7ff02a1f5fff  libsmime3.so  ???
	0x7ff02a3fa000 - 0x7ff02a41efff  libnssutil3.so  ???
	0x7ff02a626000 - 0x7ff02a75afff  libnss3.so  ???
	0x7ff02a962000 - 0x7ff02a964fff  libgmodule-2.0.so.0.4002.0  ???
	0x7ff02ab66000 - 0x7ff02ab6afff  libXtst.so.6.1.0  ???
	0x7ff02ad6c000 - 0x7ff02ad74fff  libXrender.so.1.3.0  ???
	0x7ff02af76000 - 0x7ff02af84fff  libXi.so.6.1.0  ???
	0x7ff02b186000 - 0x7ff02b18afff  libXfixes.so.3.1.0  ???
	0x7ff02b38c000 - 0x7ff02b39cfff  libXext.so.6.4.0  ???
	0x7ff02b59e000 - 0x7ff02b59ffff  libXdamage.so.1.1.0  ???
	0x7ff02b7a1000 - 0x7ff02b7a9fff  libXcursor.so.1.0.2  ???
	0x7ff02b9ab000 - 0x7ff02b9acfff  libXcomposite.so.1.0.0  ???
	0x7ff02bbae000 - 0x7ff02bbcafff  libxcb.so.1.1.0  ???
	0x7ff02bdcd000 - 0x7ff02bdcdfff  libX11-xcb.so.1.0.0  ???
	0x7ff02bfcf000 - 0x7ff02c0fefff  libX11.so.6.3.0  ???
	0x7ff02c304000 - 0x7ff02c409fff  libglib-2.0.so.0.4002.0  ???
	0x7ff02c60c000 - 0x7ff02c65afff  libgobject-2.0.so.0.4002.0  ???
	0x7ff02c85d000 - 0x7ff02c962fff  libcairo.so.2.11301.0  ???
	0x7ff02cb68000 - 0x7ff02cbb1fff  libpango-1.0.so.0.3600.3  ???
	0x7ff02cdb5000 - 0x7ff02cdbffff  libpangocairo-1.0.so.0.3600.3  ???
	0x7ff02cfc2000 - 0x7ff02cfc8fff  librt-2.19.so  ???
	0x7ff02d1ca000 - 0x7ff02d1ccfff  libdl-2.19.so  ???
	0x7ff02d3ce000 - 0x7ff02d3e6fff  libpthread-2.19.so  ???  (WARNING: No symbols, libpthread-2.19.so, 860606AF75813607E0524DF17D5A86CB0)
	0x7ff02d5ec000 - 0x7ff02d60efff  ld-2.19.so  ???
	0x7ff02d7b9000 - 0x7ff02d7c0fff  libEGL.so  ???
	0x7ff02d7ec000 - 0x7ff02d80bfff  .com.google.Chrome.m5vZG0 (deleted)  ???
	0x7ff02d811000 - 0x7ff03433cfff  chrome  ???  (main)
	0x7ffe3f37e000 - 0x7ffe3f37ffff  linux-gate.so  ???
********************************************************************************
Standard output:
********************************************************************************
	[22001:22001:0426/003054.348627:ERROR:sandbox_linux.cc(344)] InitializeSandbox() called with multiple threads in process gpu-process.
********************************************************************************
System log:
(Not implemented)
********************************************************************************
(/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:482 _ConvertExceptionFromInspectorWebsocket) Original exception:
timed out
********************************************************************************
(/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:503 _AddDebuggingInformation) Received a socket error in the browser connection and the tab still exists. The operation probably timed out.
********************************************************************************
(/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome_inspector/inspector_backend.py:504 _AddDebuggingInformation) Debugger url: ws://127.0.0.1:41530/devtools/page/43982de1-f4ce-4381-b104-9d2ca7dc2b4c
********************************************************************************
(/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome/tab_list_backend.py:111 _HandleDevToolsConnectionError) The browser exists and can be reached. The devtools target probably crashed.

Locals:
  context_id : u'43982de1-f4ce-4381-b104-9d2ca7dc2b4c'
  e          : TimeoutException('',)

(WARNING) 2017-04-26 00:38:35,097 desktop_browser_backend.Close:623  Failed to gracefully shutdown.
(WARNING) 2017-04-26 00:38:35,097 desktop_browser_backend.Close:627  Proceed to kill the browser.
[  FAILED  ] ../../../chrome/test/data/perf/canvas_bench/many_images.html (460992 ms)
 
Owner: junov@chromium.org
Assigning to benchmark owner junov. I plan to disable while we wait for bisect.
Cc: kbr@chromium.org
See Ken since the stack trace is related to webGl & GPU stuffs.
Project Member

Comment 4 by bugdroid1@chromium.org, Apr 26 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0452df84ef52ad9e638559713d18b4d85f4d93e9

commit 0452df84ef52ad9e638559713d18b4d85f4d93e9
Author: sullivan <sullivan@chromium.org>
Date: Wed Apr 26 17:59:49 2017

Disable failing smoothness.tough_canvas_cases benchmark on linux.

BUG= 715607 

Review-Url: https://codereview.chromium.org/2840233002
Cr-Commit-Position: refs/heads/master@{#467379}

[modify] https://crrev.com/0452df84ef52ad9e638559713d18b4d85f4d93e9/tools/perf/benchmarks/smoothness.py

Comment 5 by kbr@chromium.org, Apr 26 2017

Cc: xlai@chromium.org junov@chromium.org zakerinasab@chromium.org
Components: Blink>Canvas Internals>GPU>Internals
CC'ing Canvas folks.

I scanned through the regression range and don't see any obvious suspects like the enabling of accelerated 2D canvas for more canvas sizes.

My best guess is that GPU acceleration for 2D canvas was enabled on Linux recently, and so these tests now require a real GPU to run. My guess is that one isn't available and the GPU info collector is crashing (though it should be more robust to this).

Is there a command line for easy reproduction? e.g. launch xvfb and Chrome within it, running these benchmarks?

Comment 6 by kbr@chromium.org, Apr 26 2017

Labels: OS-Linux
The best way to reproduce replicate the command in https://chromium-swarm.appspot.com/task?id=35c190db608aec10&refresh=10&show_raw=1
Project Member

Comment 8 by 42576172...@developer.gserviceaccount.com, Apr 26 2017

Cc: csashi@google.com
Owner: csashi@google.com

=== Auto-CCing suspected CL author csashi@google.com ===

Hi csashi@google.com, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Test failure found with culprit

Suspected Commit
  Author : csashi
  Commit : de1410d69989d28e98ca3bc1d31a18d74dae5352
  Date   : Thu Apr 13 22:14:01 2017
  Subject: UKM that threads together multiple form interaction events.

Bisect Details
  Configuration: linux_perf_bisect
  Benchmark    : smoothness.tough_canvas_cases
  Metric       : frame_times/.._.._.._chrome_test_data_perf_canvas_bench_many_images.html

Revision             Exit Code      N
chromium@464449      0 +- N/A       20      good
chromium@464552      0 +- N/A       20      good
chromium@464559      0 +- N/A       20      good
chromium@464562      0 +- N/A       20      good
chromium@464564      0 +- N/A       20      good
chromium@464565      1 +- N/A       20      bad       <--
chromium@464578      1 +- N/A       20      bad
chromium@464604      1 +- N/A       20      bad
chromium@464655      1 +- N/A       20      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=.........chrome.test.data.perf.canvas.bench.many.images.html smoothness.tough_canvas_cases

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8981235483573861584

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=6062103980408832


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!

Comment 9 by csashi@google.com, Apr 26 2017

Hi,
My initial reaction is this regression is unlikely to be caused by my additional logging because my change will be triggered only when the user is filling out a form.

Comment 10 by csashi@google.com, Apr 26 2017

Owner: junov@chromium.org
Hi,
Since the test does not have a form, please reassign. I have filed the wrong bisect.
Thanks,
-sashi.
I kicked off another bisect.
Project Member

Comment 13 by 42576172...@developer.gserviceaccount.com, Apr 26 2017

Cc: jeffschiller@google.com
Owner: jeffschiller@google.com

=== Auto-CCing suspected CL author jeffschiller@google.com ===

Hi jeffschiller@google.com, the bisect results pointed to your CL, please take a look at the
results.


=== BISECT JOB RESULTS ===
Test failure found with culprit

Suspected Commit
  Author : jeffschiller
  Commit : 9c4df02e620f303e9e3f65818323bc3dc2dc29aa
  Date   : Fri Apr 14 00:47:38 2017
  Subject: Make several methods of TemplateUrlService const

Bisect Details
  Configuration: linux_perf_bisect
  Benchmark    : smoothness.tough_canvas_cases
  Metric       : frame_times/.._.._.._chrome_test_data_perf_canvas_bench_many_images.html

Revision             Exit Code      N
chromium@464449      0 +- N/A       20      good
chromium@464552      0 +- N/A       20      good
chromium@464604      0 +- N/A       20      good
chromium@464617      0 +- N/A       20      good
chromium@464624      0 +- N/A       20      good
chromium@464627      0 +- N/A       20      good
chromium@464629      0 +- N/A       20      good
chromium@464630      1 +- N/A       20      bad       <--
chromium@464655      1 +- N/A       20      bad

To Run This Test
  src/tools/perf/run_benchmark -v --browser=release --output-format=chartjson --upload-results --pageset-repeat=1 --also-run-disabled-tests --story-filter=.........chrome.test.data.perf.canvas.bench.many.images.html smoothness.tough_canvas_cases

Debug Info
  https://chromeperf.appspot.com/buildbucket_job_status/8981222829621161664

Is this bisect wrong?
  https://chromeperf.appspot.com/bad_bisect?try_job_id=5058310128009216


| O O | Visit http://www.chromium.org/developers/speed-infra/perf-bug-faq
|  X  | for more information addressing perf regression bugs. For feedback,
| / \ | file a bug with component Speed>Bisection.  Thank you!
Owner: ----
I reported this as a bad bisect.  Jeff's change adds "const" to a few methods for working with search engines in the browser; I can't see how it could cause the kinds of problems described here.

Multiple bad bisects suggests to me that the test results are too variable for reliable auto-bisecting in this case.

Comment 15 by kbr@chromium.org, Apr 26 2017

Owner: kbr@chromium.org
Status: Assigned (was: Available)
Writing up an analysis.

Comment 16 by kbr@chromium.org, Apr 26 2017

Cc: capn@chromium.org fs...@chromium.org sugoi@chromium.org
Components: Internals>GPU>SwiftShader
Status: WontFix (was: Assigned)
There were two tests reported failing in smoothness.tough_canvas_cases on https://build.chromium.org/p/chromium.perf/builders/Linux%20Perf :

* file://../../../chrome/test/data/perf/canvas_bench/many_images.html
* http://ie.microsoft.com/testdrive/Performance/SpeedReading/Default.html

SpeedReading was timing out. many_images was reporting the crash in libEGL.so!egl::MakeCurrent.

The last good build was:
https://build.chromium.org/p/chromium.perf/builders/Linux%20Perf/builds/567

and the first bad build:
https://build.chromium.org/p/chromium.perf/builders/Linux%20Perf/builds/568

There is no GPU on this bot; this rack-mounted server has only a built-in Matrox GPU which is not used. These tests are run with SwiftShader, which is not useful because they're not testing the behavior of accelerated 2D canvas on users' machines, the majority of which have real GPUs.

For this reason the command line invocation from:
https://chromium-swarm.appspot.com/task?id=35c190db608aec10&refresh=10&show_raw=1

isn't useful to try to reproduce locally, since on any reasonable workstation the tests will run on-screen, using the GPU.

I installed the dependencies (xcompmgr specifically; xvfb-run and openbox were already on my workstation) for our xvfb.py script and ran these:

1)
./testing/xvfb.py ./testing/scripts/run_telemetry_benchmark_as_googletest.py tools/perf/run_benchmark smoothness.tough_canvas_cases -v --output-format=chartjson --isolated-script-test-chartjson-output=/tmp/chartjson-output.json --isolated-script-test-output=/tmp/output.json --also-run-disabled-tests --story-filter=SpeedReading --browser=exact --browser-executable=/home/Downloads/chrome-linux-464554/chrome

2)
./testing/xvfb.py ./testing/scripts/run_telemetry_benchmark_as_googletest.py tools/perf/run_benchmark smoothness.tough_canvas_cases -v --output-format=chartjson --isolated-script-test-chartjson-output=/tmp/chartjson-output.json --isolated-script-test-output=/tmp/output.json --also-run-disabled-tests --story-filter=SpeedReading --browser=exact --browser-executable=/home/Downloads/chrome-linux-464657/chrome

(paths were changed slightly from the real ones)

There is no significant performance difference between these two builds.

With a top-of-tree Release build with dcheck_always_on=true, I also ran:
./testing/xvfb.py ./testing/scripts/run_telemetry_benchmark_as_googletest.py tools/perf/run_benchmark smoothness.tough_canvas_cases -v --output-format=chartjson --isolated-script-test-chartjson-output=/tmp/chartjson-output.json --isolated-script-test-output=/tmp/output.json --browser=release --also-run-disabled-tests --story-filter=many_images

It did not reproduce the libEGL crash (presumably inside SwiftShader) seen on these bots.

At this point given that this bot was never testing the true hardware accelerated path I don't think it's worth investigating this further since neither the timeout of SpeedReading nor the crash in many_images are reproducible locally. Closing as WontFix.

Status: Started (was: WontFix)
Thanks Ken for the analysis. Though the bots are still red. Shouldn't we at least disable the bot on this platform due to the lack of GPU?

Comment 18 by kbr@chromium.org, Apr 26 2017

Status: WontFix (was: Started)
These benchmarks were disabled on Linux in https://chromium.googlesource.com/chromium/src.git/+/0452df84ef52ad9e638559713d18b4d85f4d93e9 in comment #4 above. I had to pass --also-run-disabled-tests to get them to run locally. I assume the bots just haven't picked up that revision yet. Feel free to reopen if that's incorrect.

That's really weird. We should either get real GPU-accelerated canvas, or CPU (Skia) canvas. SwiftShader should only be used for WebGL, AFAIK.

Not a big deal for these tests, but it would good to know that this isn't something which will affect actual Chrome users.

Comment 20 by capn@chromium.org, Apr 27 2017

senorblanco@, note that the crash is happening within CollectGraphicsInfoGL(). So it's not actually be trying to use SwiftShader, it's just collecting info.

It's crashing due to an illegal instruction (SIGILL), but it's not caused by SwiftShader's code itself. The process is being killed with fire. Note the "Failed to gracefully shutdown" and "Proceed to kill the browser" messages.
OK, that isn't as bad as I feared, then. Should still be fixed, though, no? kbr@: WDYT? should we reopen this bug or open another one, or is this only tickled by xvfb and we don't care?
Cc: cwallez@chromium.org

Comment 23 by kbr@chromium.org, Apr 27 2017

The crash inside libEGL.so is not reproducible locally, even running under xvfb. This is the reason I don't think it is worth spending more time on this issue.

On Linux, Telemetry's graceful browser shutdown is broken, so those warnings about needing to forcibly kill the browser are common. I thought there was a bug filed about this but can't find it.

Sign in to add a comment