Adding a video that shows a reproduction of the segfault.

Deleted: Segfault_Reproduction.mp4 9.5 MB

	Segfault_Reproduction.mp4 9.5 MB View Download

The following includes relevant log info from the moment the segfault occurred:


From /var/log/ui/ui.LATEST:
[9781:9781:0126/154230.028306:ERROR:input_method_manager_impl.cc(1031)] IMEEngine for "jkghodnilhceideoidjikpgommlajknk" is not registered
device-enumerator: scan all dirs
  device-enumerator: scanning /sys/bus
device-enumerator: scan all dirs
  device-enumerator: scanning /sys/bus
  device-enumerator: scanning /sys/class
device-enumerator: scan all dirs
  device-enumerator: scanning /sys/bus
  device-enumerator: scanning /sys/class
  device-enumerator: scanning /sys/class


From /var/log/messages:
2018-01-26T16:15:03.012999-08:00 INFO kernel: [ 5051.277612] chrome[13970]: segfault at 2b8 ip 000056e61d004d77 sp 00007ffe99a17d98 error 4 in chrome[56e617fff000+86f2000]
2018-01-26T16:15:03.024305-08:00 INFO crash_reporter[14481]: libminijail[14481]: mount /dev/log -> /dev/log type ''
2018-01-26T16:15:03.036582-08:00 WARNING crash_reporter[14481]: [user] Received crash notification for chrome[13970] sig 11, user 1000 (ignoring call by kernel - chrome crash; waiting for chrome to call us directly)
2018-01-26T16:15:03.135360-08:00 INFO session_manager[8813]: [INFO:child_exit_handler.cc(77)] Handling 13970 exit.
2018-01-26T16:15:03.135674-08:00 ERR session_manager[8813]: [ERROR:child_exit_handler.cc(85)]   Exited with signal 11
2018-01-26T16:15:03.135844-08:00 INFO session_manager[8813]: [INFO:session_manager_service.cc(307)] Exiting process is chrome.
2018-01-26T16:15:03.136088-08:00 INFO session_manager[8813]: [INFO:browser_job.cc(156)] Terminating process group: Ensuring browser processes are gone.
2018-01-26T16:15:03.136336-08:00 INFO session_manager[8813]: [INFO:system_utils_impl.cc(94)] Sending 9 to -13970 as 1000
2018-01-26T16:15:03.413018-08:00 INFO session_manager[8813]: [INFO:browser_job.cc(147)] Running child /opt/google/chrome/chrome --ppapi-flash-path=/opt/google/chrome/pepper/libpepflashplayer.so --ppapi-flash-version=28.0.0.152 --ui-prioritize-in-gpu-process --use-gl=egl --enable-native-gpu-memory-buffers --enable-drm-atomic --enable-hardware-overlays=single-fullscreen,single-on-top --enable-webgl-image-chromium --enable-features=Pepper3DImageChromium --gpu-sandbox-failures-fatal=yes --enable-logging --log-level=1 --use-cras --enable-wayland-server --user-data-dir=/home/chronos --max-unused-resource-memory-usage-percentage=5 --system-developer-mode --login-profile=user --has-chromeos-keyboard --enable-touchview --guest-wallpaper-large=/usr/share/chromeos-assets/wallpaper/guest_large.jpg --guest-wallpaper-small=/usr/share/chromeos-assets/wallpaper/guest_small.jpg --child-wallpaper-large=/usr/share/chromeos-assets/wallpaper/child_large.jpg --child-wallpaper-small=/usr/share/chromeos-assets/wallpaper/child_small.jpg --default-wallpaper-large=/usr/share/chromeos-assets/wallpaper/default_large.jpg --default-wallpaper-small=/usr/share/chromeos-assets/wallpaper/default_small.jpg --arc-availability=officially-supported --enterprise-enrollment-initial-modulus=15 --enterprise-enrollment-modulus-limit=19 --login-user=casey.g.bowman@intel.corp-partner.google.com --login-profile=a24d4f003ad7a8d25792e8de934f91d1fd4ff39a --vmodule=*arc/*=1,automatic_reboot_manager=1,tablet_power_button_controller=1,*chromeos/login/*=1,auto_enrollment_controller=1,*plugin*=2,*zygote*=1,*/ui/ozone/*=1,*/ui/display/manager/chromeos/*=1,*night_light*=1,power_button_observer=2,webui_login_view=2,lock_state_controller=2,webui_screen_locker=2,screen_locker=2 --enable-features=Pepper3DImageChromium

CC+=Vladislav

Please help re-assign if possible. Thanks for the helps.

Why am I cc'd on this bug?

Sorry I am not sure I reached out to the right person.

CC+=Stéphane as well.

Thanks.

Seems  like a UI problem -> Albert should be able to triage

Logs look like a chrome crash. Are there any IDs in chrome://crashes ?

Given the repro steps I'm going to guess it's something with the network dialogs, but it would be good to get an ID so we can get a stack.

@7
Unfortunately, no, there weren't any IDs listed. I turned on the crash reporting feature, reproduced the issue, but I didn't see any entries within chrome://crashes.

Thanks. Can you take a look in chrome://system and share the value from the CLIENT_ID row?

CLIENT_ID: aac89bea-807b-4f47-8264-97c5dc152422

Odd. I'm not seeing any reports at all from a machine with that ID in the crash system. It's worrisome that it looks like this is crashing without giving us anything to work with.

+stevenjb any thoughts here?

Chrome crash reporting is one area where I haven't really retained much helpful knowledge.

I think I recall a change where we can no longer search by Client Id?

+derat@, +ihf@, +jamescook@

We should have some folks try to repro this locally since there is a repro, even if it is a bit tricky.

Not aware of changes. Searching by CLIENT_ID still works, example
https://crash.corp.google.com/browse?q=ClientID%3D%27b8e11e723b2d4dfdbae99cc62475224f%27
But maybe there is some non-trivial translation going on? Or the crashes are indeed lost.

Re: #8 - is chrome://crashes still empty on that device? They get queued up to send. Even if you click on "Start uploading crashes" it takes a minute or two before the list populates.

Also, I don't think we upload crashes from machines with "test" images. I have these notes on how to force uploads:

metrics_client -C  # force metrics consent
touch /run/crash_reporter/crash-test-in-progress # make crash_sender try to upload when running a test image
crash_sender -e FORCE_OFFICIAL=1 -e SECONDS_SEND_SPREAD=1
grep crash_sender /var/log/messages

I think there are some (old) notes here: http://www.chromium.org/chromium-os/packages/crash-reporting 

If you still have the device with the crashes, maybe try those command line steps?

Re searching by client ID, issue 668153 tracks the formats of IDs varying in different places. I don't know if it's still an issue, but I've switched to searching for crashes with "like '[initial hex digits]%'" to work around it ('like' is case-insensitive, and just using the initial digits avoids the dashes being there or not).

RE: #14

It looks like each of those given methods work, as chrome://crashes has 3 separate crashes logged (because why not test them all?).

Here is the first crash report ID:
5536fa49013ebbf2

That crash Id was actually for a kernal crash, which may or may not be related, but from that I was able to get your client id and identify two crashes with a call stack that macthes the described sumptom (stack below).

+jennyz@ who has also recently been in the Tray code, but I will try to investigate this myself since it does appear to be in NetworkStateListDetailedView.


	0x000058597bbb0c17	(chrome -network_state_list_detailed_view.cc:313 )	non-virtual thunk to ash::tray::NetworkStateListDetailedView::InfoBubble::OnMouseExited(ui::MouseEvent const&)
0x000058597a30b111	(chrome -event_dispatcher.cc:191 )	ui::EventDispatcher::ProcessEvent(ui::EventTarget*, ui::Event*)
0x000058597a30af1b	(chrome -event_dispatcher.cc:86 )	ui::EventDispatcherDelegate::DispatchEvent(ui::EventTarget*, ui::Event*)
0x000058597b16aaa9	(chrome -root_view.cc:696 )	views::internal::RootView::NotifyEnterExitOfDescendant(ui::MouseEvent const&, ui::EventType, views::View*, views::View*)
0x000058597b16abfd	(chrome -root_view.cc:568 )	views::internal::RootView::OnMouseExited(ui::MouseEvent const&)
0x000058597b140793	(chrome -widget.cc:1248 )	views::Widget::OnMouseEvent(ui::MouseEvent*)
0x000058597a30b111	(chrome -event_dispatcher.cc:191 )	ui::EventDispatcher::ProcessEvent(ui::EventTarget*, ui::Event*)
0x000058597a30af1b	(chrome -event_dispatcher.cc:86 )	ui::EventDispatcherDelegate::DispatchEvent(ui::EventTarget*, ui::Event*)
0x000058597adafd45	(chrome -window_event_dispatcher.cc:311 )	aura::WindowEventDispatcher::DispatchMouseEnterOrExit(aura::Window*, ui::MouseEvent const&, ui::EventType)
0x000058597adb067a	(chrome -window_event_dispatcher.cc:179 )	aura::WindowEventDispatcher::DispatchMouseExitToHidingWindow(aura::Window*)
0x000058597adaaca3	(chrome -window.cc:762 )	aura::Window::SetVisible(bool)
0x000058597b14aa9a	(chrome -native_widget_aura.cc:509 )	views::NativeWidgetAura::Close()
0x000058597b13e6e1	(chrome -widget.cc:601 )	views::Widget::Close()
0x000058597bbaecbe	(chrome -network_state_list_detailed_view.cc:316 )	extensions::Action::set_page_incognito(bool)
0x000058597bbaa79d	(chrome -network_list.cc:528 )	std::__1::__tuple_impl<std::__1::__tuple_indices<0ul, 1ul, 2ul, 3ul>, base::WeakPtr<net::WebSocketTransportClientSocketPool>, net::ClientSocketHandle*, base::RepeatingCallback<void (int)>, int>::~__tuple_impl()
0x000058597b133b63	(chrome -memory:2399 )	views::View::DoRemoveChildView(views::View*, bool, bool, bool, views::View*)
0x000058597b134024	(chrome -view.cc:300 )	views::View::RemoveAllChildViews(bool)
0x000058597baafb93	(chrome -system_tray_bubble.cc:132 )	ash::SystemTrayBubble::UpdateView(std::__1::vector<ash::SystemTrayItem*, std::__1::allocator<ash::SystemTrayItem*> > const&, ash::SystemTrayView::SystemTrayType)
0x000058597baadc18	(chrome -system_tray.cc:451 )	ash::SystemTray::ShowItems(std::__1::vector<ash::SystemTrayItem*, std::__1::allocator<ash::SystemTrayItem*> > const&, bool, ash::BubbleCreationType, bool, bool)
0x000058597baadb53	(chrome -system_tray.cc:327 )	std::__1::allocator_traits<std::__1::allocator<char> >::allocate(std::__1::allocator<char>&, unsigned long)
0x000058597bb6713c	(chrome -tray_details_view.cc:456 )	ash::TrayDetailsView::DoTransitionToDefaultView()
0x0000585979930062	(chrome -callback.h:94 )	base::Timer::RunScheduledTask()
0x000058597e24eb64	(chrome -callback.h:65 )	base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*)
0x000058597e25084e	(chrome -message_loop.cc:399 )	base::MessageLoop::RunTask(base::PendingTask*)
0x000058597e251018	(chrome -message_loop.cc:411 )	base::MessageLoop::DoDelayedWork(base::TimeTicks*)
0x000058597e25143a	(chrome -message_pump_libevent.cc:230 )	base::MessagePumpLibevent::Run(base::MessagePump::Delegate*)
0x000058597990bc44	(chrome -run_loop.cc:130 )	mojo::internal::Array_Data<char>::Array_Data(unsigned int, unsigned int)
0x00005859795c6d07	(chrome -chrome_browser_main.cc:1959 )	ChromeBrowserMainParts::MainMessageLoopRun(int*)
0x000058597816a3a3	(chrome -browser_main_loop.cc:1245 )	content::BrowserMainLoop::RunMainMessageLoopParts()
0x000058597816d331	(chrome -browser_main_runner.cc:145 )	content::BrowserMainRunnerImpl::Run()
0x0000585978166348	(chrome -browser_main.cc:46 )	content::BrowserMain(content::MainFunctionParams const&)
0x00005859795b36e0	(chrome -content_main_runner.cc:712 )	content::ContentMainRunnerImpl::Run()
0x00005859795bc83b	(chrome -main.cc:456 )	service_manager::Main(service_manager::MainParams const&)
0x00005859795b23a0	(chrome -content_main.cc:19 )	content::ContentMain(content::ContentMainParams const&)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/be1d327bfd98a0e2cb258398a808e048c8e4c648

commit be1d327bfd98a0e2cb258398a808e048c8e4c648
Author: Steven Bennetts <stevenjb@chromium.org>
Date: Wed Feb 14 19:50:53 2018

Network Tray: Protect from edge case crash

Bug:  806446 
Change-Id: Ibb34f1ea9c43315f61cf21a8680123f1d3bc6f21
Reviewed-on: https://chromium-review.googlesource.com/914847
Commit-Queue: Steven Bennetts <stevenjb@chromium.org>
Reviewed-by: Jenny Zhang <jennyz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#536783}
[modify] https://crrev.com/be1d327bfd98a0e2cb258398a808e048c8e4c648/ash/system/network/network_state_list_detailed_view.cc

@Steve

Would you happen to know the first browser version that includes the fix?

I pulled down the latest image from today and was still able to reproduce the segfault.


Device Info:
  Board: reef
     OS: 10409.0.0 (Official Build) dev-channel reef-unibuild (reef pyro snappy sand electro basking alan bigdaddy) test
 Chrome: Google Chrome 66.0.3344.0 unknown
     FW: Google_Reef.9042.110.0
     EC: reef_v1.1.5909-bd1f0c9
    CPU: Intel(R) Celeron(R) CPU N3350 @ 1.10GHz (2 cores)
 Kernel: Linux 4.4.115-13105-g4a1cbdf295af x86_64
    Mem: 4GB (physical)

Fixed in Chrome 66.0.3349.0

There hasn't been a Chrome update on canary all week due to a combinaiton of infra issues and build issues. Hopefully we'll get one out next week.

No problem. When it lands in the dev channel, I'll do some post-merge testing and report back.

I pulled down an image with Chrome 66.0.3352.0 and attempted to reproduce the issue for about 5 minutes.

My attempts failed and I couldn't reproduce the issue; the post-merge image looks good!