New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 884804 link

Starred by 2 users
Status: Fixed
Owner:
enne@chromium.org
Closed: Oct 22
Cc:
█ bsalomon@chromium.org
enne@chromium.org
khushals...@chromium.org
zmo@chromium.org
Components:
EstimatedDays: ----
NextAction: 2018-10-22
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

ChromeOS OOP-R Mali crashes

Project Member Reported by enne@chromium.org, Sep 17 
I see two OOP-R crashers on ChromeOS.  They are both in libmali.  The GPU vendor and GPU device ID are all bogus, but they are all driver version "22.0.01rel0" and almost all Linux 3.14.0.*.  Not a ton of reports yet, but CPM is 3-4, which seems like something we should probably address vs could ignore.

https://crash.corp.google.com/browse?q=product_name%3D%22Chrome_ChromeOS%22+AND+expanded_custom_data.ChromeCrashProto.magic_signature_1.name%3D%22%5BGPU+hang%5D+GrGLGpu%3A%3AdeleteFramebuffer%22#-propertyselector,-productname:1000,-magicsignature:50,-magicsignature2:50,-stablesignature:50,gpuvendorid,gpudeviceid,gpudriverversion,+experiments,-magicsignaturesorted:50

Thread 0 (id: 0x1331) MAGIC SIGNATURE THREAD
Stack Quality100%Show frame trust levels
0xaa720514	(libpthread-2.23.so + 0x0000e514 )	__libc_do_syscall
0xaa71d85b	(libpthread-2.23.so -futex-internal.h:205 )	do_futex_wait
0xaa71d93d	(libpthread-2.23.so -sem_waitcommon.c:282 )	__new_sem_wait_slow
0xaa71d99d	(libpthread-2.23.so -sem_wait.c:29 )	__new_sem_wait
0xa8eaaa2f	(libmali.so.0.1.20 -mali_osu_temp_release.h:135 )	cframep_context_inc_used_resources
0xa8ebf8cd	(libmali.so.0.1.20 -mali_cframe_tilelist.c:1738 )	cframep_tilelist_reset
0xa8ebf59d	(libmali.so.0.1.20 -mali_cframe_tilelist.c:1296 )	cframep_tilelist_orphan
0xa8eadc2b	(libmali.so.0.1.20 -mali_cframe_manager.c:592 )	cframep_manager_delete
0xa9044299	(libmali.so.0.1.20 -mali_gles_fb_object_lx.c:357 )	glesx_fbp_object_delete
0xa9047a6f	(libmali.so.0.1.20 -mali_gles_fb_object.c:190 )	gles_fbp_object_delete
0xa904495d	(libmali.so.0.1.20 -mali_gles_fb_api.c:143 )	gles_fb_delete_framebuffers
0x0a4c6747	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/include/gpu/gl/GrGLFunctions.h:311 )	GrGLGpu::deleteFramebuffer(unsigned int)
0x0a4d42e3	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/src/gpu/gl/GrGLRenderTarget.cpp:163 )	GrGLRenderTarget::onRelease()
0x0a4d58bf	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/src/gpu/gl/GrGLTextureRenderTarget.h:49 )	virtual thunk to GrGLTextureRenderTarget::onRelease()
0x0a46610f	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/src/gpu/GrGpuResource.cpp:54 )	GrGpuResource::release()
0x0a477777	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/src/gpu/GrGpuResourceCacheAccess.h:36 )	GrResourceCache::purgeUnlockedResources(bool)
0x07dd73ef	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/base/callback.h:99 )	void base::internal::CancelableCallbackImpl<base::OnceCallback<void ()> >::ForwardOnce<>()
0x07f282e7	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/base/callback.h:99 )	base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*)
0x09c8cad3	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/base/message_loop/message_loop.cc:434 )	scoped_refptr<base::internal::WeakReference::Flag const>::scoped_refptr(scoped_refptr<base::internal::WeakReference::Flag const>&&)
0x09c8d023	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/base/message_loop/message_loop.cc:445 )	base::MessageLoop::DoDelayedWork(base::TimeTicks*)
0x07f1dd41	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/base/message_loop/message_pump_default.cc:41 )	base::MessagePumpDefault::Run(base::MessagePump::Delegate*)
0x09ca2237	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/base/run_loop.cc:102 )	std::__1::__compressed_pair<unsigned int, std::__1::allocator<std::__1::__hash_node_base<std::__1::__hash_node<std::__1::__hash_value_type<base::MemoryPressureListener*, scoped_refptr<base::SequencedTaskRunner> >, void*>*>*> >::__compressed_pair<int, true>(int&&)
0x0c440405	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/content/gpu/gpu_main.cc:347 )	content::GpuMain(content::MainFunctionParams const&)
0x099da495	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/content/app/content_main_runner_impl.cc:563 )	content::ContentMainRunnerImpl::Run(bool)
0x099df7a1	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/services/service_manager/embedder/main.cc:472 )	service_manager::Main(service_manager::MainParams const&)
0x099d957f	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/content/app/content_main.cc:19 )	content::ContentMain(content::ContentMainParams const&)
0x07ffafcf	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/chrome/app/chrome_main.cc:101 )	ChromeMain


https://crash.corp.google.com/browse?q=expanded_custom_data.ChromeCrashProto.magic_signature_1.name%3D%22%5BGPU+hang%5D+GrGLGpuRTCommandBuffer%3A%3Abegin%22+AND+product_name%3D%27Chrome_ChromeOS%27#-propertyselector,-productname:1000,-magicsignature:50,-magicsignature2:50,-stablesignature:50,gpuvendorid,gpudeviceid,gpudriverversion,+experiments,-magicsignaturesorted:50

Thread 0 (id: 0x60b) MAGIC SIGNATURE THREAD
Stack Quality100%Show frame trust levels
0xb0cc3514	(libpthread-2.23.so + 0x0000e514 )	__libc_do_syscall
0xb0cc085b	(libpthread-2.23.so -futex-internal.h:205 )	do_futex_wait
0xb0cc093d	(libpthread-2.23.so -sem_waitcommon.c:282 )	__new_sem_wait_slow
0xb0cc099d	(libpthread-2.23.so -sem_wait.c:29 )	__new_sem_wait
0xaf62267f	(libmali.so.0.1.20 -mali_osu_temp_release.h:135 )	cframep_context_inc_used_resources
0xaf63751d	(libmali.so.0.1.20 -mali_cframe_tilelist.c:1738 )	cframep_tilelist_reset
0xaf6280c1	(libmali.so.0.1.20 -mali_cframe_manager.c:6124 )	cframep_prepare_next_tilelist_common
0xaf629e39	(libmali.so.0.1.20 -mali_cframe_manager.c:8202 )	cframep_manager_reset
0xaf62bdc1	(libmali.so.0.1.20 -mali_cframe_manager.c:9953 )	cframep_manager_clear_full
0xaf62a197	(libmali.so.0.1.20 -mali_cframe_manager.c:9690 )	cframe_manager_clear
0xaf4298eb	(libmali.so.0.1.20 -mali_gles_fb_helpers_lx.c:31 )	glesx_fbp_clear
0xaf42f62d	(libmali.so.0.1.20 -mali_gles_fb_helpers.c:227 )	gles_fbp_clear
0xaf42b63b	(libmali.so.0.1.20 -mali_gles_fb_api.c:1335 )	gles_fb_clear
0x08e186fd	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/src/gpu/gl/GrGLGpuCommandBuffer.cpp:15 )	GrGLGpuRTCommandBuffer::begin()
0x08dbb72f	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/src/gpu/GrRenderTargetOpList.cpp:163 )	GrRenderTargetOpList::onExecute(GrOpFlushState*)
0x08da8ddb	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/include/private/GrOpList.h:40 )	GrDrawingManager::internalFlush(GrSurfaceProxy*, GrResourceCache::FlushType, int, GrBackendSemaphore*)
0x08da94ed	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/src/gpu/GrDrawingManager.h:97 )	GrDrawingManager::prepareSurfaceForExternalIO(GrSurfaceProxy*, int, GrBackendSemaphore*)
0x08dba0ab	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/src/gpu/GrRenderTargetContext.cpp:1401 )	GrRenderTargetContext::prepareForExternalIO(int, GrBackendSemaphore*)
0x0b46bbcb	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/third_party/skia/src/image/SkSurface.cpp:234 )	gpu::raster::RasterDecoderImpl::HandleEndRasterCHROMIUM(unsigned int, void const volatile*)
0x0b4716e1	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/gpu/command_buffer/service/raster_decoder.cc:1544 )	gpu::raster::RasterDecoderImpl::DoCommands(unsigned int, void const volatile*, int, int*)
0x0c2a70bb	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/gpu/command_buffer/service/command_buffer_service.cc:69 )	gpu::CommandBufferService::Flush(int, gpu::AsyncAPIInterface*)
0x0b4934d3	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/gpu/ipc/service/command_buffer_stub.cc:621 )	gpu::CommandBufferStub::OnMessageReceived(IPC::Message const&)
0x0b49a841	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/gpu/ipc/service/gpu_channel.cc:533 )	gpu::GpuChannel::HandleMessageHelper(IPC::Message const&)
0x0b4989a5	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/gpu/ipc/service/gpu_channel.cc:509 )	gpu::GpuChannel::HandleMessage(IPC::Message const&)
0x06916815	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/base/callback.h:99 )	gpu::Scheduler::RunNextTask()
0x06864d25	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/base/callback.h:99 )	base::MessageLoop::DoWork()
0x06865d2b	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/base/message_loop/message_pump_default.cc:37 )	base::MessagePumpDefault::Run(base::MessagePump::Delegate*)
0x085ea237	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/base/run_loop.cc:102 )	std::__1::__compressed_pair<unsigned int, std::__1::allocator<std::__1::__hash_node_base<std::__1::__hash_node<std::__1::__hash_value_type<base::MemoryPressureListener*, scoped_refptr<base::SequencedTaskRunner> >, void*>*>*> >::__compressed_pair<int, true>(int&&)
0x0ad883d5	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/content/gpu/gpu_main.cc:347 )	content::GpuMain(content::MainFunctionParams const&)
0x08322495	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/content/app/content_main_runner_impl.cc:563 )	content::ContentMainRunnerImpl::Run(bool)
0x083277a1	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/services/service_manager/embedder/main.cc:472 )	service_manager::Main(service_manager::MainParams const&)
0x0832157f	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/content/app/content_main.cc:19 )	content::ContentMain(content::ContentMainParams const&)
0x06942fcf	(chrome -./../../../../../../../home/chrome-bot/chrome_root/src/chrome/app/chrome_main.cc:101 )	ChromeMain

Comment 1 by enne@chromium.org, Sep 24 

I looked through about ~70 of the top crash to see if there were different gpus involved, and they were 100% Mali T-760.  I'm going to just disable OOP-R for this config.

Comment 2 by enne@chromium.org, Sep 24

Cc: zmo@chromium.org
zmo points out that these are all gpu hang detections, and so a better solution is to add something similar to ContextGroup::ReportProgress() to Skia.
Project Member

Comment 3 by bugdroid1@chromium.org, Oct 3 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2

commit a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2
Author: Adrienne Walker <enne@chromium.org>
Date: Wed Oct 03 01:53:22 2018

gpu: Add Skia plumbing to watch dog thread

Now that Skia is running GL directly as a part of OOP-R, it means that
the keep alive mechanisms for the command buffer are no longer being
applied.  If any part of Skia runs long, then the watch dog hang
detector will kill the gpu process.

In particular, the gpu command buffer does this via:
* ContextGroup::ReportProgress
* GLES2DecoderImpl::ReportProgress
* GLES2DecoderImpl::ExitCommandProcessingEarly

...which all wind up eventually in GpuWatchdogThread::ReportProgress.

This fixes this by hooking up CreateGrGLInterface to ReportProgress on several "slow" calls.

Bug:  884804 , 887939
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_chromium_dbg_ng;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Change-Id: I84f2dc92670de3f1dd1ca7e0e8aa85c2d08a4858
Reviewed-on: https://chromium-review.googlesource.com/c/1242033
Commit-Queue: enne <enne@chromium.org>
Reviewed-by: Antoine Labour <piman@chromium.org>
Reviewed-by: Zhenyao Mo <zmo@chromium.org>
Cr-Commit-Position: refs/heads/master@{#596091}
[modify] https://crrev.com/a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2/build/check_gn_headers_whitelist.txt
[modify] https://crrev.com/a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2/gpu/command_buffer/service/context_group.cc
[modify] https://crrev.com/a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2/gpu/command_buffer/service/context_group.h
[modify] https://crrev.com/a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2/gpu/command_buffer/service/program_manager.cc
[modify] https://crrev.com/a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2/gpu/command_buffer/service/program_manager.h
[delete] https://crrev.com/523814ca611efd11917e03e8ad3c2212ea6f043d/gpu/command_buffer/service/progress_reporter.h
[modify] https://crrev.com/a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2/gpu/command_buffer/service/raster_decoder_context_state.cc
[modify] https://crrev.com/a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2/gpu/command_buffer/service/raster_decoder_context_state.h
[modify] https://crrev.com/a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2/gpu/command_buffer/service/shader_manager.cc
[modify] https://crrev.com/a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2/gpu/command_buffer/service/shader_manager.h
[modify] https://crrev.com/a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2/gpu/command_buffer/service/texture_manager.cc
[modify] https://crrev.com/a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2/gpu/command_buffer/service/texture_manager.h
[modify] https://crrev.com/a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2/gpu/ipc/service/gpu_channel_manager.cc
[modify] https://crrev.com/a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2/gpu/ipc/service/gpu_watchdog_thread.h
[modify] https://crrev.com/a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2/ui/gl/BUILD.gn
[modify] https://crrev.com/a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2/ui/gl/init/create_gr_gl_interface.cc
[modify] https://crrev.com/a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2/ui/gl/init/create_gr_gl_interface.h
[add] https://crrev.com/a237d180c9ca68a68fae1e83c1d06ebcf46ce6a2/ui/gl/progress_reporter.h

Comment 4 by enne@chromium.org, Oct 10

NextAction: 2018-10-22

Comment 5 by monor...@bugs.chromium.org, Oct 22 

The NextAction date has arrived: 2018-10-22

Comment 6 by enne@chromium.org, Oct 22

Status: Fixed (was: Assigned)
This seems fixed, based on crash reports.  This landed in 71.0.3570.0, and the last crashes are from 71.0.3567.0.
►

Sign in to add a comment