Logs present here: https://pantheon.corp.google.com/storage/browser/chromiumos-test-logs/bugfiles/cr/874616

We have chameleon test display_HotPlugAtBoot.mirrored failing on these boards since R70-10962.0.0 ( issue 874562 ). With this build chrome has updated to v.70.0.3519.3

There are many different crashes, but one I see often in the above listed IDs is this:

Thread 0 (id: 0x423) CRASHED [SIGILL / ILL_ILLOPN @ 0x00005d2ce8e41700 ] MAGIC SIGNATURE THREAD
Stack Quality100%Show frame trust levels
0x00005d2ce8e41700	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/content/browser/compositor/owned_mailbox.cc:48 )	content::OwnedMailbox::Destroy()
0x00005d2ce8e3eda4	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/content/browser/compositor/gpu_process_transport_factory.cc:737 )	content::GpuProcessTransportFactory::RemoveCompositor(ui::Compositor*)
0x00005d2cebba6200	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/ui/compositor/compositor.cc:246 )	ui::Compositor::~Compositor()
0x00005d2cebba6ead	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/ui/compositor/compositor.cc:227 )	<name omitted>
0x00005d2cebb9801f	(chrome -memory:2321 )	aura::WindowTreeHostPlatform::~WindowTreeHostPlatform()
0x00005d2ceca6bd69	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/ash/host/ash_window_tree_host_platform.cc:51 )	<name omitted>
0x00005d2ceca85880	(chrome -memory:2321 )	ash::RootWindowController::~RootWindowController()
0x00005d2ceca56858	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/ash/display/window_tree_host_manager.cc:225 )	ash::WindowTreeHostManager::Shutdown()
0x00005d2cecaad2cb	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/ash/shell.cc:916 )	ash::Shell::~Shell()
0x00005d2cecaaebbd	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/ash/shell.cc:730 )	ash::Shell::~Shell()
0x00005d2cecf808fd	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/ash/shell.cc:298 )	ChromeBrowserMainExtraPartsAsh::PostMainMessageLoopRun()
0x00005d2cea22604d	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/chrome/browser/chrome_browser_main.cc:2094 )	ChromeBrowserMainParts::PostMainMessageLoopRun()
0x00005d2ce9474cdc	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/chrome/browser/chromeos/chrome_browser_main_chromeos.cc:1155 )	chromeos::ChromeBrowserMainPartsChromeos::PostMainMessageLoopRun()
0x00005d2ce88ee5f6	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/content/browser/browser_main_loop.cc:1066 )	content::BrowserMainLoop::ShutdownThreadsAndCleanUp()
0x00005d2ce88f182d	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/content/browser/browser_main_runner_impl.cc:217 )	content::BrowserMainRunnerImpl::Shutdown()
0x00005d2ce88e66ec	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/content/browser/browser_main.cc:49 )	content::BrowserMain(content::MainFunctionParams const&)
0x00005d2cea211acd	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/content/app/content_main_runner_impl.cc:536 )	content::ContentMainRunnerImpl::Run(bool)
0x00005d2cea2197db	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/services/service_manager/embedder/main.cc:472 )	service_manager::Main(service_manager::MainParams const&)
0x00005d2ce7bc7dc4	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/content/app/content_main.cc:19 )	ChromeMain
0x0000702729bd7735	(libc-2.23.so -libc-start.c:289 )	__libc_start_main
0x00005d2ce7bb8978	(chrome + 0x002ab978 )	_start


Kyle this might be related to this change: https://chromium-review.googlesource.com/c/chromium/src/+/1150456. Can you please take a look?

It wouldn't be https://crrev.com/c/1150456, there is no behaviour change unless running with --enable-features=VizDisplayCompositor and OwnedMailbox doesn't work with that feature enabled. It sounds like it started in this range?

https://chromium.googlesource.com/chromium/src/+log/70.0.3511.0..70.0.3519.3?pretty=fuller&n=10000

OwnedMailbox is accessing a gpu::gles2::GLES2Interface* in Destroy(). The GLES2Interface is owned by a ref counted ContextProvider. The OwnedMailbox is ref counted and is owned by both ReflectorImpl and ReflectorTexture. ReflectorImpl, ReflectorTexture and ContextProvider are all owned by GpuBrowserCompositorOutputSurface.

The problem appears to be that a scoped_refptr<OwnedMailbox> can exist on a callback when GpuBrowserCompositorOutputSurface is destroyed. The OwnedMailbox doesn't get deleted because of that scoped_refptr, but the GLES2Interface does get deleted. I'm not sure exactly what changed but the shutdown order stuff here is a bit of a mess.  When the last reference disappears 

I think this must have worked at some point because ContextFactoryObserver::ReleaseResources(), which is now OnLostSharedContext(), was getting called on shutdown? Changing the destruction order for a bunch of things could impact this.

Looking at the OwnedMailbox/ReflectorImpl/ReflectorTexture the ownership is confusing. As far as I can tell, GpuProcessTransportFactory creates a ReflectorImpl. The ReflectorImpl gets installed in GpuBrowserCompositorOutputSurface, which creates a ReflectorTexture and OwnedMailbox, and also gives a reference to the OwnedMailbox to ReflectorImpl.

I think in ReflectorImpl::DetachFromOutputSurface() when we drops the reference to the OwnedMailbox we could also just destroy the OwnedMailbox? Also OwnedMailbox doesn't need to be a ContextFactoryObserver anymore. I'd need to look closer to verify.

afakhry: I tested using Ozone X11 and https://crrev.com/c/1178221 seems to fix the crashes. I don't have a CrOS device handy to test with now though.

Observed on today's build on Peach-Pi - 70.0.3524.2/10985.0.0 as soon as the HDMI cable plugged into DUT.

2018-08-20T22:32:37.394294+00:00 NOTICE crash_sender.sh[6043]: Crash report receipt ID 3b29377759088c56
2018-08-20T22:32:39.580818+00:00 NOTICE crash_sender.sh[6212]: Crash report receipt ID c7937da30969a971
2018-08-20T22:32:42.347812+00:00 NOTICE crash_sender.sh[6467]: Crash report receipt ID 639944b3f689f35d
2018-08-20T22:32:44.587143+00:00 NOTICE crash_sender.sh[6648]: Crash report receipt ID 976409876712c16c
2018-08-20T22:32:47.129391+00:00 NOTICE crash_sender.sh[6874]: Crash report receipt ID 2f6097a0e264d1c5
2018-08-20T22:32:50.507333+00:00 NOTICE crash_sender.sh[7081]: Crash report receipt ID 89a79005248f55c6
2018-08-20T22:32:53.171445+00:00 NOTICE crash_sender.sh[7249]: Crash report receipt ID d94bde28ebbf28a8
2018-08-20T22:32:55.246951+00:00 NOTICE crash_sender.sh[7424]: Crash report receipt ID f2a7aa8e619e181b
2018-08-20T22:32:57.571044+00:00 NOTICE crash_sender.sh[7609]: Crash report receipt ID 4fd16754dfa6c7ea
2018-08-20T22:32:59.969160+00:00 NOTICE crash_sender.sh[7776]: Crash report receipt ID 5156747caac75cf0
2018-08-20T22:33:01.984659+00:00 NOTICE crash_sender.sh[7950]: Crash report receipt ID 1c24a76edccd45d7
2018-08-20T15:35:47.143549-07:00 NOTICE crash_sender.sh[3759]: Crash report receipt ID 8ff966232ef22427
2018-08-20T15:35:49.067774-07:00 NOTICE crash_sender.sh[3865]: Crash report receipt ID f1e58dd66682e377
2018-08-20T15:35:50.891090-07:00 NOTICE crash_sender.sh[3994]: Crash report receipt ID 1be275d28f2fd0a4
2018-08-20T15:35:55.060456-07:00 NOTICE crash_sender.sh[4186]: Crash report receipt ID 39798b7cd8ff9f1a
2018-08-20T15:35:57.545141-07:00 NOTICE crash_sender.sh[4422]: Crash report receipt ID cf262cc08621bdcb
2018-08-20T15:35:59.974425-07:00 NOTICE crash_sender.sh[4599]: Crash report receipt ID a86f84169d60cca4
2018-08-20T15:36:02.940381-07:00 NOTICE crash_sender.sh[4780]: Crash report receipt ID 80d99fdecef75622
2018-08-20T15:36:05.112027-07:00 NOTICE crash_sender.sh[4966]: Crash report receipt ID 01f231be5d6f645f
2018-08-20T15:36:07.524218-07:00 NOTICE crash_sender.sh[5139]: Crash report receipt ID 810f78e8fd1fc023
2018-08-20T15:36:10.403193-07:00 NOTICE crash_sender.sh[5320]: Crash report receipt ID 119dc5badee4749e
2018-08-20T15:36:13.110999-07:00 NOTICE crash_sender.sh[5487]: Crash report receipt ID bc628937190cb887
2018-08-20T15:36:15.517286-07:00 NOTICE crash_sender.sh[5670]: Crash report receipt ID a09a2252d7407b01
2018-08-20T15:36:17.584679-07:00 NOTICE crash_sender.sh[5851]: Crash report receipt ID 3e77290fd9d5243d
2018-08-20T15:36:19.885879-07:00 NOTICE crash_sender.sh[6032]: Crash report receipt ID 43090ed430f58bbd
2018-08-20T15:36:22.808630-07:00 NOTICE crash_sender.sh[6258]: Crash report receipt ID 622ce377000fd8cf
2018-08-20T15:36:25.070724-07:00 NOTICE crash_sender.sh[6424]: Crash report receipt ID 61cdacc474a40c44

Several signatures here but most commonly I'm seeing bug 876385

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/628ffc023972b08bffe84bf3f816fb47a535c736

commit 628ffc023972b08bffe84bf3f816fb47a535c736
Author: kylechar <kylechar@chromium.org>
Date: Wed Aug 22 13:08:30 2018

Fix OwnedMailbox use-after-free on destruction.

OwnedMailbox is ref counted and can outlive the GLES2Interface* it
holds. This is because the callback has a scoped_refptr<OwnedMailbox>
which prevents destroyed the OwnedMailbox.

The ownership model for OwnedMailbox should probably be reworked, but
the class will be replaced fairly soon anyways. Change callback to hold
a WeakPtr instead of scoped_refptr so that OwnedMailbox gets destroyed
at the correct time.

OwnedMailbox is also no longer used with GLHelper and the shared main
thread context, so it doesn't need to be a ContextFactoryObserver to
find out about losing the shared main thread context. Delete
ImageTransportFactoryTearDownBrowserTest.LoseOnTearDown which tested the
now deleted functionality.

Bug:  874616 
Change-Id: Iab95a906c4006427e0a0046c56fe20f75d9788a6
Reviewed-on: https://chromium-review.googlesource.com/1178221
Commit-Queue: kylechar <kylechar@chromium.org>
Reviewed-by: ccameron <ccameron@chromium.org>
Cr-Commit-Position: refs/heads/master@{#584991}
[modify] https://crrev.com/628ffc023972b08bffe84bf3f816fb47a535c736/content/browser/compositor/image_transport_factory_browsertest.cc
[modify] https://crrev.com/628ffc023972b08bffe84bf3f816fb47a535c736/content/browser/compositor/owned_mailbox.cc
[modify] https://crrev.com/628ffc023972b08bffe84bf3f816fb47a535c736/content/browser/compositor/owned_mailbox.h
[modify] https://crrev.com/628ffc023972b08bffe84bf3f816fb47a535c736/content/browser/compositor/reflector_impl.cc

afakhry: Can you verify if this is fixed?

Friendly ping to get an update as it is marked as RBS. Thanks

kyle, sorry, I didn't have a chance to give your CL a try. I'll test it later today.

I don't have any of the devices mentioned in #0. pgangishetty@ Please verify that issue is now fixed after commit in #9.

Not reproducible on Auron_paine with build version 71.0.3544.0/11053.0.0 

Thanks for checking if this occurs still.

[Auto-generated comment by a script] We noticed that this issue is targeted for M-70; it appears the fix may have landed after branch point, meaning a merge might be required. Please confirm if a merge is required here - if so add Merge-Request-70 label, otherwise remove Merge-TBD label. Thanks.

#9 landed in M70.

Not reproducible on Auron_paine, Cyan & Peppy with build version 70.0.3538.15/11021.12.0.  Marked as verified.