Issue metadata
Sign in to add a comment
|
chrome --mash fails on device |
||||||||||||||||||||
Issue descriptionmustash Chrome crashes on start. Am bisect-ing. Dump. localhost chrome # sudo -u chronos ./chrome --user-data-dir=/home/chronos --homedir=/ --mash [12121:12126:0331/120436.990173:1737994235:INFO:service_process_launcher.cc(196)] Launched child process pid=12127, instance=, name=content_packaged_services, user_id=d701c6a2-66cd-47b2-8ce3-bb6022aa34d8 [12121:12126:0331/120436.993261:1737997161:INFO:service_process_launcher.cc(196)] Launched child process pid=12128, instance=70453656-3234-485e-a2ea-45fd57356923, name=chrome_mash, user_id=505C0EE9-3013-43C0-82 B0-A84F50CF8D84 [12121:12126:0331/120436.994587:1737998486:INFO:service_process_launcher.cc(196)] Launched child process pid=12129, instance=3fe10287-3305-403e-b837-5125a4568310, name=chrome_mash, user_id=505C0EE9-3013-43C0-82 B0-A84F50CF8D84 [12121:12126:0331/120436.996010:1737999909:INFO:service_process_launcher.cc(196)] Launched child process pid=12130, instance=209e0ae2-7b7b-440a-bbcb-ab7308dd1542, name=chrome_mash, user_id=c57352e6-019d-4eb6-a9 e8-6bc35e8a7bad [12121:12124:0331/120437.032106:1738035996:ERROR:instance.cc(71)] Unable to locate service manifest for tracing [12121:12124:0331/120437.032176:1738036062:ERROR:service_manager.cc(947)] Failed to resolve service name: tracing [12121:12124:0331/120437.034325:1738038214:ERROR:instance.cc(71)] Unable to locate service manifest for tracing [12121:12124:0331/120437.034373:1738038259:ERROR:service_manager.cc(947)] Failed to resolve service name: tracing device-enumerator: scan all dirs device-enumerator: scanning /sys/bus device-enumerator: scanning /sys/class [12130:12147:0331/120437.080142:1738084077:ERROR:vaapi_wrapper.cc(1212)] Could not get a valid VA display [12121:12126:0331/120437.088665:1738092565:INFO:service_process_launcher.cc(196)] Launched child process pid=12153, instance=7843f7a4-7dca-4b52-b53d-df3175cbb981, name=chrome_mash, user_id=505C0EE9-3013-43C0-82 B0-A84F50CF8D84 [12121:12124:0331/120437.122138:1738126027:ERROR:instance.cc(71)] Unable to locate service manifest for tracing [12121:12124:0331/120437.122189:1738126075:ERROR:service_manager.cc(947)] Failed to resolve service name: tracing [12121:12124:0331/120437.131107:1738134997:ERROR:instance.cc(71)] Unable to locate service manifest for tracing [12121:12124:0331/120437.131179:1738135066:ERROR:service_manager.cc(947)] Failed to resolve service name: tracing [12128:12128:0331/120437.182745:1738186633:ERROR:device_event_log_impl.cc(137)] [12:04:37.182] Network: network_handler_callbacks.cc:84 not-supported: /device/wlan0: org.chromium.flimflam.Error.NotSupported: Th is WiFi device does not support MAC address randomization [12130:12130:0331/120438.239337:1739243234:ERROR:screen_manager_ozone_internal.cc(321)] Not implemented reached in virtual void display::ScreenManagerOzoneInternal::CloseMirroringDisplayIfNotNecessary() [12128:12128:0331/120438.245295:1739249187:ERROR:shell_delegate_mus.cc(131)] Not implemented reached in virtual std::unique_ptr<PaletteDelegate> ash::ShellDelegateMus::CreatePaletteDelegate() [12128:12128:0331/120438.245531:1739249420:ERROR:wm_shell_mus.cc(378)] Not implemented reached in virtual void ash::mus::WmShellMus::AddDisplayObserver(ash::WmDisplayObserver *) [12128:12128:0331/120438.245742:1739249631:ERROR:shell_delegate_mus.cc(87)] Not implemented reached in virtual void ash::ShellDelegateMus::PreInit() [12128:12128:0331/120438.247375:1739251265:ERROR:wm_shell_mus.cc(276)] Not implemented reached in virtual void ash::mus::WmShellMus::RecordUserMetricsAction(ash::UserMetricsAction) [12128:12128:0331/120438.249184:1739253075:ERROR:shell_delegate_mus.cc(121)] Not implemented reached in virtual ash::SessionStateDelegate *ash::ShellDelegateMus::CreateSessionStateDelegate() device-enumerator: scan all dirs device-enumerator: scanning /sys/bus device-enumerator: scanning /sys/class [12130:12130:0331/120438.276754:1739280643:ERROR:screen_manager_ozone_internal.cc(321)] Not implemented reached in virtual void display::ScreenManagerOzoneInternal::CloseMirroringDisplayIfNotNecessary() [12128:12128:0331/120438.282282:1739286170:ERROR:wm_shell_mus.cc(221)] Not implemented reached in virtual bool ash::mus::WmShellMus::IsInUnifiedMode() const [12128:12128:0331/120438.284527:1739288415:ERROR:object_proxy.cc(583)] Failed to call method: org.chromium.PowerManager.GetSwitchStates: object_path= /org/chromium/PowerManager: org.freedesktop.DBus.Error.Unkn$wnMethod: Method "GetSwitchStates" with signature "" on interface "org.chromium.PowerManager" doesn't exist [12128:12128:0331/120438.284590:1739288475:ERROR:device_event_log_impl.cc(137)] [12:04:38.284] Power: power_manager_client.cc:563 Error calling GetSwitchStates [12128:12128:0331/120438.297495:1739301384:ERROR:layer_tree_host_impl.cc(2260)] Forcing zero-copy tile initialization as worker context is missing [12128:12128:0331/120438.313401:1739317290:ERROR:wm_shell_mus.cc(77)] Not implemented reached in virtual bool ash::mus::(anonymous namespace)::SessionStateDelegateStub::ShouldShowAvatar(ash::WmWindow *) const [12128:12128:0331/120438.318777:1739322667:ERROR:shell_delegate_mus.cc(77)] Not implemented reached in virtual bool ash::ShellDelegateMus::CanShowWindowForUser(ash::WmWindow *) const [12127:12127:0331/120438.396691:ERROR:cras_audio_handler.cc(690)] Unexpexted OutputNodeVolumeChanged received on node: 0x500000001 [12127:12127:0331/120438.397983:ERROR:object_proxy.cc(583)] Failed to call method: org.chromium.SessionManagerInterface.GetServerBackedStateKeys: object_path= /org/chromium/SessionManager: org.freedesktop.DBus$Error.ServiceUnknown: The name org.chromium.SessionManager was not provided by any .service files [12127:12127:0331/120438.398019:ERROR:session_manager_client.cc(705)] Failed to call GetServerBackedStateKeys [12127:12127:0331/120438.421764:ERROR:remote_text_input_client.cc(140)] Not implemented reached in virtual void RemoteTextInputClient::OnInputMethodChanged() [12127:12127:0331/120438.460220:ERROR:device_event_log_impl.cc(137)] [12:04:38.460] Network: network_handler_callbacks.cc:84 not-supported: /device/wlan0: org.chromium.flimflam.Error.NotSupported: This WiFi de$ice does not support MAC address randomization [12129:12129:0331/120438.479139:1739483095:ERROR:layer_tree_host_impl.cc(2260)] Forcing zero-copy tile initialization as worker context is missing [1:5:0331/120438.479611:ERROR:layer_tree_host_impl.cc(2260)] Forcing zero-copy tile initialization as worker context is missing [12127:12127:0331/120438.481871:ERROR:layer_tree_host_impl.cc(2260)] Forcing zero-copy tile initialization as worker context is missing Received signal 11 SEGV_MAPERR 000000000010 #0 0x7f6d9c52739c base::debug::StackTrace::StackTrace() #1 0x7f6d9c526f11 base::debug::(anonymous namespace)::StackDumpSignalHandler() #2 0x7f6d99791580 <unknown> #3 0x7f6d9aa7ffa3 content::ServiceManagerConnectionImpl::IOThreadContext::CreateService() #4 0x7f6d9ab7d325 service_manager::mojom::ServiceFactoryStubDispatch::Accept() #5 0x7f6d9c5cd3e8 mojo::InterfaceEndpointClient::HandleValidatedMessage() #6 0x7f6d9c5d29b8 mojo::internal::MultiplexRouter::ProcessIncomingMessage() #7 0x7f6d9c5d2416 mojo::internal::MultiplexRouter::Accept() #8 0x7f6d9c5cc26d mojo::Connector::ReadSingleMessage() #9 0x7f6d9c5cc7e1 mojo::Connector::ReadAllAvailableMessages() #10 0x7f6d9c5da360 mojo::SimpleWatcher::OnHandleReady() #11 0x7f6d9c5da5ab mojo::SimpleWatcher::Context::Notify() #12 0x7f6d9c5d9cd1 mojo::SimpleWatcher::Context::CallNotify() #13 0x7f6d9d4d197e mojo::edk::Watch::InvokeCallback() #14 0x7f6d9d4cffce mojo::edk::RequestContext::~RequestContext() #15 0x7f6d9d4d8867 mojo::edk::NodeChannel::OnChannelMessage() #16 0x7f6d9d4d4dde mojo::edk::Channel::OnReadComplete() #17 0x7f6d9d4d5fdd mojo::edk::(anonymous namespace)::ChannelPosix::OnFileCanReadWithoutBlocking() #18 0x7f6d9c543029 base::MessagePumpLibevent::OnLibeventNotification() #19 0x7f6d9c5c4605 event_base_loop #20 0x7f6d9c54324e base::MessagePumpLibevent::Run() #21 0x7f6d9c54125b base::MessageLoop::RunHandler() #22 0x7f6d9c560b70 base::RunLoop::Run() #23 0x7f6d9ae3fac8 content::BrowserThreadImpl::IOThreadRun() #24 0x7f6d9ae3fc8f content::BrowserThreadImpl::Run() #25 0x7f6d9c582e88 base::Thread::ThreadMain() #26 0x7f6d9c57e40d base::(anonymous namespace)::ThreadFunc() #27 0x7f6d99787558 <unknown> #28 0x7f6d984b208d clone r8: 0000000000000007 r9: c6a4a7935bd1e995 r10: 89133354f2050967 r11: 0000000000000000 r12: 00000000000001ed r13: 00007f6d98c91220 r14: 000032c3432cbd60 r15: 00007f6da2400da0 di: 000000000000000b si: 0000000000000003 bp: 00007f6d8c8472c0 bx: 00007f6d8c8472d0 dx: 0000000000000007 ax: 0000000000000000 cx: 00000000000001ed sp: 00007f6d8c8472a0 ip: 00007f6d9aa7ffa3 efl: 0000000000010287 cgf: 0000000000000033 erf: 0000000000000004 trp: 000000000000000e msk: 0000000000000000 cr2: 0000000000000010 [end of stack trace] Calling _exit(1). Core file will not be generated. [12121:12121:0331/120438.707020:1739710910:ERROR:mash_runner.cc(162)] Main process exiting because service content_packaged_services quit unexpectedly. [12128:12128:0331/120438.708005:1739711895:ERROR:wm_shell_mus.cc(383)] Not implemented reached in virtual void ash::mus::WmShellMus::RemoveDisplayObserver(ash::WmDisplayObserver *) [12128:12128:0331/120438.709579:1739713470:ERROR:shell_delegate_mus.cc(91)] Not implemented reached in virtual void ash::ShellDelegateMus::PreShutdown() ../../third_party/tcmalloc/chromium/src/free_list.h:118] Memory corruption detected. Received signal 11 SEGV_MAPERR 000000000039 #0 0x7f07e39f139c base::debug::StackTrace::StackTrace() #1 0x7f07e39f0f11 base::debug::(anonymous namespace)::StackDumpSignalHandler() #2 0x7f07e0c5b580 <unknown> #3 0x7f07e1e134f0 <unknown> #4 0x7f07e1e177ab tcmalloc::Log() #5 0x7f07e1e16f44 tcmalloc::FL_PopRange() #6 0x7f07e1e1b9e1 tcmalloc::ThreadCache::ReleaseToCentralCache() #7 0x7f07e1e1c336 tcmalloc::ThreadCache::DeleteCache() #8 0x7f07e0c50538 <unknown> #9 0x7f07e0c5161d <unknown> #10 0x7f07df97c08d clone r8: 0000000000000000 r9: cccccccccccccccd r10: 00007f07e8002a80 r11: 0000000000000000 r12: 00007f07dbf3cd01 r13: 00007f07dbf3cc3b r14: 00007f07dbf3cc08 r15: 0000000000000001 di: 0000000000000000 si: 00007f07dbf3cc08 bp: 00007f07dbf3cd20 bx: 0000000000000056 dx: 0000000000000056 ax: 0000000000000000 cx: 00007f07e0c5a66d sp: 00007f07dbf3cbd8 ip: 00007f07e1e134f0 efl: 0000000000010297 cgf: 0000000000000033 erf: 0000000000000006 trp: 000000000000000e msk: 0000000000000000 cr2: 0000000000000039 [end of stack trace] Calling _exit(1). Core file will not be generated.
,
Mar 31 2017
Probably would hit this DCHECK: https://cs.chromium.org/chromium/src/content/common/service_manager/service_manager_connection_impl.cc?rcl=d8b95960a8f2bec0e0eb48e636ee57204804bad9&l=333 I suppose that should be a LOG(ERROR) and an early return rather than a DCHECK and possible nullptr deref.
,
Apr 1 2017
per bisection, it's https://codereview.chromium.org/2772673002 that introduced this crash when running on link (aka pixel) devices.
,
Apr 2 2017
,
Apr 2 2017
I'm looking into this. It looks like my service isn't getting registered under this configuration for some reason.
Could I have some more repro steps please? I'm assuming from the crash and Ken's comment that this isn't a debug build. This also works fine on my Linux machine if I run:
ninja -j2000 -C out/ChromeOS mash:all chrome && out/ChromeOS/chrome --enable-logging=stderr --mash
with the build settings
use_goma=true
is_component_build=true
is_debug=false
use_ozone=true
target_os="chromeos"
enable_wayland_server=true
dcheck_always_on=true
enable_nacl=false
Should I be getting a device to test on or should I be able to repro this on a Linux machine?
,
Apr 2 2017
I'm guess PrefServiceEnabled returns false in whatever configuration this was built under: https://cs.chromium.org/chromium/src/chrome/common/chrome_features.cc?sq=package:chromium&dr&l=316 Is ENABLE_PACKAGE_MASH_SERVICES enabled in this context?
,
Apr 3 2017
ENABLE_PACKAGE_MASH_SERVICES is enabled by default when target_os == chromeos
,
Apr 3 2017
,
Apr 3 2017
The bug replicates on both samus and link CrOS devices. The build configuration was the default for building CrOS Chrome for a device per simple-chrome.
,
Apr 3 2017
If you're not familiar with simplechrome the strip_flags instructions here make it easier to get backtraces on device in /var/log/ui/ui.LATEST without a full debug build. https://yaqs.googleplex.com/eng/q/5267421445488640
,
Apr 3 2017
So nothing strikes me as immediately out of place. Though if ENABLE_PACKAGE_MASH_SERVICES is on for cros builds, this makes me wonder if the mash flag is not set by the time that the pref service is attempting to register itself. I won't have access to a device to debug this on until the morning. So I'm uploading a CL to disable ash from attempting to connect to the missing services. To unblock mash work in the interim. https://codereview.chromium.org/2795863002/
,
Apr 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/fe52253d07c2dfed2fc3cb60e6bc2baf5e386327 commit fe52253d07c2dfed2fc3cb60e6bc2baf5e386327 Author: jonross <jonross@chromium.org> Date: Mon Apr 03 22:04:26 2017 Disable Shell from connecting to prefs service Currently, when using --mash, ash/shell is unable to connect to the prefs service which is provided by chrome. This change disables attempting to connect while in --mash mode. I will reenable once the root cause is found. TEST=ran chrome --mash to verify no boot crash BUG= 707321 Review-Url: https://codereview.chromium.org/2795863002 Cr-Commit-Position: refs/heads/master@{#461551} [modify] https://crrev.com/fe52253d07c2dfed2fc3cb60e6bc2baf5e386327/ash/shell.cc
,
Apr 4 2017
Jonathan, I'm assigning this to you for now. Let me know if you need me to take over the debugging (in which case I need to know how to repro in more detail, since I don't work on mash).
,
Apr 4 2017
Issue 622127 has been merged into this issue.
,
Apr 4 2017
So I've done some on device debugging. ENABLE_PACKAGE_MASH_SERVICES is true. By the chrome_features::PrefServiceEnabled is called, the mash flag is appropriately set. So we do not have a race here. chrome_content_browser_client is successfully registering its service. Even with the ash connection to the pref service disabled we are still seeing this crash. I had to revert the suspect change in #3 to get mash working. Though this revert is no longer clean, there have been changes since. I'm reassigning tibell@ as the change itself will need more debugging. As mentioned above this only occurs on device, right now Link is the most stable device for mash. The standard method for making a local builds is "simple chrome" https://sites.google.com/a/chromium.org/dev/chromium-os/how-tos-and-troubleshooting/building-chromium-browser?pli=1 For stack traces to work properly you will need to roll the debug version, which will require some of these extra steps in order to fit on device: https://sites.google.com/a/chromium.org/dev/chromium-os/how-tos-and-troubleshooting/building-chromium-browser?pli=1#TOC-Debug-build Once you've pushed a build you can enable mash by editing /etc/chrome_dev.conf on device. # Add these lines to the bottom --mash !--login-manager If you have questions feel free to ping me, I've done lots on device before.
,
Apr 4 2017
Actually it appears that on Link, with my disabling, that we can get past this issue. While there was not a race in the preconditions for registering the pref service I have seen a race between chrome and ash/shell.cc. On desktop, using --mash, the shell is completing init before the pref service is registered. This seems to lead to the request silently being dropped. Though delaying the request does lead to a successful connection. On device (Link in this case), the shell was starting after the registration of the pref service. Though on device we see the associated crash.
,
Apr 4 2017
I think this is a race condition and CreateService() should queue requests for services that haven't been registered yet.
,
Apr 4 2017
,
Apr 5 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/26b9dcec30000075deb5bf3455e99961c4db8b17 commit 26b9dcec30000075deb5bf3455e99961c4db8b17 Author: sky <sky@chromium.org> Date: Wed Apr 05 00:57:54 2017 Makes ServiceManagerConnectionImpl queue requests Without this it's possible for CreateService() to be called before the request handler. BUG= 707321 TEST=none R=ben@chromium.org Review-Url: https://codereview.chromium.org/2795413002 Cr-Commit-Position: refs/heads/master@{#461925} [modify] https://crrev.com/26b9dcec30000075deb5bf3455e99961c4db8b17/content/common/service_manager/service_manager_connection_impl.cc
,
Apr 6 2017
With https://codereview.chromium.org/2795413002 landed can we try to enable the pref service in mash again?
,
Apr 6 2017
As to Ken's prodding I realized that fix is wrong. I moved the discussion to email. Let me know if you have more questions.
,
Apr 6 2017
Based on offline discussions I'm going to take another look at this. We want to make sure that ash connects to the correct profile. Currently the Off the Record profile is not registering a pref services. We also want ash to be able to handle profile switching, and to properly reconnect.
,
Apr 10 2017
I landed https://codereview.chromium.org/2802093005/ which resolves the startup crash on mash. |
|||||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||||
Comment 1 by jamescook@chromium.org
, Mar 31 2017