Inability to debug chromeos-amd64-generic-rel specific failure |
|||||
Issue descriptionOn https://chromium-review.googlesource.com/c/chromium/src/+/1305891 : I'm getting failures only in telemetry_unittests on chromeos-amd64-generic-rel (timeouts which I'm having a hard time diagnosing). But the gn_args are ridiculous (https://pastebin.com/AmxJZqhB) and I can't get GN to even accept them locally on Linux.
,
Nov 2
Thanks for filing. The gn args are indeed quite a handful. But they're automagically filled out via the simplechrome sdk (which is what that bot uses to build): https://chromium.googlesource.com/chromiumos/docs/+/master/simple_chrome_workflow.md#enter-the-simple-chrome-environment I'll look into the failure you're seeing. The browser is likely crashing, and the tests should collect the system logs afterwards, but I'm not seeing them there. I'll look into it.
,
Nov 2
[95/96] telemetry.core.tracing_controller_unittest.StartupTracingTest.testCloseBrowserBeforeTracingIsStopped failed unexpectedly 253.2634s:
Chose browser: PossibleCrOSBrowser(browser_type=cros-chrome)
Created trace config file in /tmp/chrome-trace-config.json
(Re)starting the ui (logs the user out)
Restarting Chrome (pid=17692) with remote port
dbus-send --system --type=method_call --dest=org.chromium.SessionManager /org/chromium/SessionManager org.chromium.SessionManagerInterface.EnableChromeTesting boolean:true array:string:"--enable-net-benchmarking,--metrics-recording-only,--no-default-browser-check,--no-first-run,--ignore-background-tasks,--enable-gpu-benchmarking,--deny-permission-prompts,--autoplay-policy=no-user-gesture-required,--disable-background-networking,--disable-component-extensions-with-background-pages,--disable-default-apps,--disable-search-geolocation-disclosure,--enable-smooth-scrolling,--enable-threaded-compositing,--remote-debugging-port=0,--start-maximized,--ash-disable-system-sounds,--mute-audio,--oobe-skip-postlogin,--disable-logging-redirect,--vmodule=*/chromeos/net/*=2\,*/chromeos/login/*=2\,chrome_browser_main_posix=2,--allow-failed-policy-fetch-for-test,--disable-gaia-services,--trace-config-file=/tmp/chrome-trace-config.json" array:string:
Failed with BrowserConnectionGoneException while starting the browser backend.
*************** BROWSER STANDARD OUTPUT ***************
Cannot get standard output on CrOS
*********** END OF BROWSER STANDARD OUTPUT ************
********************* BROWSER LOG *********************
No log file
***************** END OF BROWSER LOG ******************
********************* SYMBOLIZED MINIDUMP *********************
***************** END OF SYMBOLIZED MINIDUMP ******************
Closing browser (pid=17852) ...
No devtools clients for issuing clock sync.
Remove trace config file in /tmp/chrome-trace-config.json
Created trace config file in /tmp/chrome-trace-config.json
(Re)starting the ui (logs the user out)
Browser is closed.
No devtools clients for issuing clock sync.
Remove trace config file in /tmp/chrome-trace-config.json
Traceback (most recent call last):
File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/decorators.py", line 219, in wrapper
func(*args, **kwargs)
File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/core/tracing_controller_unittest.py", line 174, in testCloseBrowserBeforeTracingIsStopped
with self.possible_browser.BrowserSession(self.browser_options) as browser:
File "/b/s/w/ir/.swarming_module/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/possible_browser.py", line 92, in BrowserSession
browser = self.Create()
File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome/cros_browser_finder.py", line 105, in Create
browser_backend, self._platform_backend, startup_args)
File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/browser/browser.py", line 48, in __init__
self._browser_backend.Start(startup_args, startup_url=startup_url)
File "/b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
return func(*args, **kwargs)
File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome/cros_browser_backend.py", line 80, in Start
self.BindDevToolsClient()
File "/b/s/w/ir/third_party/catapult/common/py_trace_event/py_trace_event/trace_event_impl/decorators.py", line 75, in traced_function
return func(*args, **kwargs)
File "/b/s/w/ir/third_party/catapult/telemetry/telemetry/internal/backends/chrome/chrome_browser_backend.py", line 118, in BindDevToolsClient
raise exceptions.BrowserConnectionGoneException(self.browser, e)
BrowserConnectionGoneException: Timed out while waiting 240s for _GetDevToolsClient.
Found Minidump: False
Stack Trace:
********************************************************************************
Cannot get stack trace on CrOS
********************************************************************************
Standard output:
********************************************************************************
Cannot get standard output on CrOS
********************************************************************************
System log:
(Not implemented)
,
Nov 2
And that test is failing due to a crashing chrome in the cros VM. (stack trace pasted below) There are docs on reproing these things locally: https://chromium.googlesource.com/chromiumos/docs/+/master/cros_vm.md#run-telemetry-unit-tests [2843:2843:1102/145330.509920:FATAL:setuid_sandbox_host.cc(157)] The SUID sandbox helper binary was found, but is not configured correctly. Rather than run without sandboxing I'm aborting now. You need to make sure that /opt/google/chrome/chrome-sandbox is owned by root and has mode 4755. #0 0x57cd95c491bf base::debug::StackTrace::StackTrace() #1 0x57cd95ba185b logging::LogMessage::~LogMessage() #2 0x57cd9a283690 sandbox::SetuidSandboxHost::PrependWrapper() #3 0x57cd975aeec3 service_manager::ZygoteHostImpl::LaunchZygote() #4 0x57cd956dfaba content::(anonymous namespace)::LaunchZygoteHelper() #5 0x57cd975ae239 service_manager::ZygoteCommunication::Init() #6 0x57cd975b0513 service_manager::CreateGenericZygote() #7 0x57cd956df164 content::ContentMainRunnerImpl::Initialize() #8 0x57cd956e6a85 service_manager::Main() #9 0x57cd956dd7e1 content::ContentMain() #10 0x57cd922700bf ChromeMain #11 0x7bed0e113736 __libc_start_main #12 0x57cd9226fee9 _start Received signal 6 #0 0x57cd95c491bf base::debug::StackTrace::StackTrace() #1 0x57cd95c48ca1 base::debug::(anonymous namespace)::StackDumpSignalHandler() #2 0x7bed0ed312e0 <unknown> #3 0x7bed0e126dd2 gsignal #4 0x7bed0e128bf6 abort #5 0x57cd95c47a45 base::debug::BreakDebugger() #6 0x57cd95ba1c86 logging::LogMessage::~LogMessage() #7 0x57cd9a283690 sandbox::SetuidSandboxHost::PrependWrapper() #8 0x57cd975aeec3 service_manager::ZygoteHostImpl::LaunchZygote() #9 0x57cd956dfaba content::(anonymous namespace)::LaunchZygoteHelper() #10 0x57cd975ae239 service_manager::ZygoteCommunication::Init() #11 0x57cd975b0513 service_manager::CreateGenericZygote() #12 0x57cd956df164 content::ContentMainRunnerImpl::Initialize() #13 0x57cd956e6a85 service_manager::Main() #14 0x57cd956dd7e1 content::ContentMain() #15 0x57cd922700bf ChromeMain #16 0x7bed0e113736 __libc_start_main #17 0x57cd9226fee9 _start r8: 0000000000000000 r9: 00007bed0dbed740 r10: 0000000000000008 r11: 0000000000000206 r12: 00007fffe4953c88 r13: 00007fffe4953c78 r14: 00007fffe4953c80 r15: 00007fffe4953411 di: 0000000000000b1b si: 0000000000000b1b bp: 00007fffe49533c0 bx: 00007fffe4953410 dx: 0000000000000006 ax: 0000000000000000 cx: 00007bed0e126dd2 sp: 00007fffe4953298 ip: 00007bed0e126dd2 efl: 0000000000000206 cgf: 002b000000000033 erf: 0000000000000000 trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000 [end of stack trace] Calling _exit(1). Core file will not be generated. [2899:2899:1102/145331.974471:FATAL:setuid_sandbox_host.cc(157)] The SUID sandbox helper binary was found, but is not configured correctly. Rather than run without sandboxing I'm aborting now. You need to make sure that /opt/google/chrome/chrome-sandbox is owned by root and has mode 4755. #0 0x56813e1dc1bf base::debug::StackTrace::StackTrace() #1 0x56813e13485b logging::LogMessage::~LogMessage() #2 0x568142816690 sandbox::SetuidSandboxHost::PrependWrapper() #3 0x56813fb41ec3 service_manager::ZygoteHostImpl::LaunchZygote() #4 0x56813dc72aba content::(anonymous namespace)::LaunchZygoteHelper() #5 0x56813fb41239 service_manager::ZygoteCommunication::Init() #6 0x56813fb43513 service_manager::CreateGenericZygote() #7 0x56813dc72164 content::ContentMainRunnerImpl::Initialize() #8 0x56813dc79a85 service_manager::Main() #9 0x56813dc707e1 content::ContentMain() #10 0x56813a8030bf ChromeMain #11 0x7926869ec736 __libc_start_main #12 0x56813a802ee9 _start Received signal 6 #0 0x56813e1dc1bf base::debug::StackTrace::StackTrace() #1 0x56813e1dbca1 base::debug::(anonymous namespace)::StackDumpSignalHandler() #2 0x79268760a2e0 <unknown> #3 0x7926869ffdd2 gsignal #4 0x792686a01bf6 abort #5 0x56813e1daa45 base::debug::BreakDebugger() #6 0x56813e134c86 logging::LogMessage::~LogMessage() #7 0x568142816690 sandbox::SetuidSandboxHost::PrependWrapper() #8 0x56813fb41ec3 service_manager::ZygoteHostImpl::LaunchZygote() #9 0x56813dc72aba content::(anonymous namespace)::LaunchZygoteHelper() #10 0x56813fb41239 service_manager::ZygoteCommunication::Init() #11 0x56813fb43513 service_manager::CreateGenericZygote() #12 0x56813dc72164 content::ContentMainRunnerImpl::Initialize() #13 0x56813dc79a85 service_manager::Main() #14 0x56813dc707e1 content::ContentMain() #15 0x56813a8030bf ChromeMain #16 0x7926869ec736 __libc_start_main #17 0x56813a802ee9 _start r8: 0000000000000000 r9: 00007926864c6740 r10: 0000000000000008 r11: 0000000000000206 r12: 00007fff486ab5a8 r13: 00007fff486ab598 r14: 00007fff486ab5a0 r15: 00007fff486aad31 di: 0000000000000b53 si: 0000000000000b53 bp: 00007fff486aace0 bx: 00007fff486aad30 dx: 0000000000000006 ax: 0000000000000000 cx: 00007926869ffdd2 sp: 00007fff486aabb8 ip: 00007926869ffdd2 efl: 0000000000000206 cgf: 002b000000000033 erf: 0000000000000000 trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000 [end of stack trace]
,
Nov 2
And https://chromium-review.googlesource.com/c/chromium/src/+/1316430 to get these logs automatically captured from the test and accessible from the builds.
,
Nov 6
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/9b24d85070cdbd78146e3724bf22cf368ab1df38 commit 9b24d85070cdbd78146e3724bf22cf368ab1df38 Author: Ben Pastene <bpastene@chromium.org> Date: Tue Nov 06 00:42:09 2018 Make gn_helpers.py safely ignore comments. I'd like to add comments to the GN args of some bots. This will let us do that without mb or simplechrome SDK crashing. Bug: 901471 Change-Id: I7f49434e02591639a02ef9cfd0f026680f6b9de1 Reviewed-on: https://chromium-review.googlesource.com/c/1318188 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#605544} [modify] https://crrev.com/9b24d85070cdbd78146e3724bf22cf368ab1df38/build/gn_helpers.py [modify] https://crrev.com/9b24d85070cdbd78146e3724bf22cf368ab1df38/build/gn_helpers_unittest.py
,
Nov 6
,
Nov 7
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/ea60cca4831d1e90195ed56785995b1673a276bf commit ea60cca4831d1e90195ed56785995b1673a276bf Author: Ben Pastene <bpastene@chromium.org> Date: Wed Nov 07 14:34:49 2018 Roll third_party/gn_helpers forward. Rolls it to 9b24d85070cdbd78146e3724bf22cf368ab1df38 to pick up https://chromium-review.googlesource.com/c/1318188 This'll let us add comments to a chromium bot's GN args, which I plan on doing for the simplechrome bots since it can confuse some chromium devs. BUG=chromium:901471 TEST=built chrome w/ simplechrome SDK Change-Id: I5b8253f9cf8c4f6811dbd8dde2ee44db71dcf2e0 Reviewed-on: https://chromium-review.googlesource.com/1320309 Commit-Ready: Ben Pastene <bpastene@chromium.org> Tested-by: Ben Pastene <bpastene@chromium.org> Reviewed-by: Achuith Bhandarkar <achuith@chromium.org> [modify] https://crrev.com/ea60cca4831d1e90195ed56785995b1673a276bf/third_party/gn_helpers/README.chromium [modify] https://crrev.com/ea60cca4831d1e90195ed56785995b1673a276bf/third_party/gn_helpers/gn_helpers.py
,
Nov 8
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/65ccf613b9cf4b98d2d8af520899bed36a24d901 commit 65ccf613b9cf4b98d2d8af520899bed36a24d901 Author: Ben Pastene <bpastene@chromium.org> Date: Thu Nov 08 00:47:59 2018 mb: Add a top-level comment to GN args when using the Simple Chrome SDK. This will add a comment to the gn_arg log link on chromium bots: https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8930669115526655984/+/steps/lookup_GN_args/0/logs/gn_args/0 This will hopefully help avoid confusion when a non-CrOS chromium dev gets a CL blocked by a simplechrome bot. (See the linked bug where someone wasn't familiar with it and tried copy-pasting the arg dump outside the SDK.) I looked into adding the comment further upstream in chromite's cros_chrome_sdk libs, but it makes heavy use of gn_helpers, and adding comment support to that is non-trivial. This can't land until https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1320309 and gets rolled into chromium. Bug: 901471 Change-Id: I9ad646a9b6ab6fd29d4adb7acfd842b26c3825ae Reviewed-on: https://chromium-review.googlesource.com/c/1318179 Reviewed-by: Steven Bennetts <stevenjb@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> Cr-Commit-Position: refs/heads/master@{#606257} [modify] https://crrev.com/65ccf613b9cf4b98d2d8af520899bed36a24d901/tools/mb/mb.py [modify] https://crrev.com/65ccf613b9cf4b98d2d8af520899bed36a24d901/tools/mb/mb_unittest.py
,
Jan 16
(6 days ago)
,
Jan 16
(6 days ago)
|
|||||
►
Sign in to add a comment |
|||||
Comment 1 by gab@chromium.org
, Nov 2