Link to all the Builds on which this crash is seen:

https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Mac%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27gpu-process%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BGPU%20hang%5D%20gfx%3A%3AGLContextCGL%3A%3ADestroy%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#-samplereports:5,productversion:1000


Crash rate on the recent M51 builds:

51.0.2692.0	0.29%	58	
51.0.2691.0	0.37%	75	
51.0.2690.0	0.42%	85	
51.0.2689.0	0.50%	100	
51.0.2688.0	0.01%	3	


Looping in chromium//src/content/common/gpu/OWNERS as i am unable to find any recent changes in the code search.

@Ken, can you please take a look at it and assign it accordingly ?

The most likely culprit is ccameron@ with:

Mac: Use AVSampleBufferDisplayLayer for 4:2:0 surfaces
https://codereview.chromium.org/1828523003

Mac: Decode hardware to 420 instead of 422
https://codereview.chromium.org/1822173002

Users experienced this crash on the following builds:

Win Dev 51.0.2687.0 - 443 reports, 406 clients (signature [GPU hang] content::DXVAVideoDecodeAccelerator::SendMFTMessage)
Mac Canary 51.0.2692.0 - 63 reports, 51 clients (signature [GPU hang] gfx::GLContextCGL::Destroy)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bd8763305febe8b7c2467734848bfd42bead244a

commit bd8763305febe8b7c2467734848bfd42bead244a
Author: ccameron <ccameron@chromium.org>
Date: Tue Mar 29 21:07:54 2016

Revert of Mac: Decode hardware to 420 instead of 422 (patchset #3 id:40001 of https://codereview.chromium.org/1822173002/ )

Reason for revert:
This likely spiked the crash rates.
BUG= 598388 

Original issue's description:
> Mac: Decode hardware to 420 instead of 422
>
> If the decoded frame is going to be used as a CALayer overlay, then
> it will never be converted to RGBA. If it is used by OpenGL, then the
> same infrastructure for software frames will be employed to make an
> expensive RGBA copy of the frame.
>
> BUG= 594452 
>
> Committed: https://crrev.com/24da601d444c8822c5ff0a10751b9384f25dd1b4
> Cr-Commit-Position: refs/heads/master@{#382797}

TBR=sandersd@chromium.org
# Not skipping CQ checks because original CL landed more than 1 days ago.
BUG= 594452 

Review URL: https://codereview.chromium.org/1842673003

Cr-Commit-Position: refs/heads/master@{#383823}

[modify] https://crrev.com/bd8763305febe8b7c2467734848bfd42bead244a/content/common/gpu/media/gpu_video_decode_accelerator.cc
[modify] https://crrev.com/bd8763305febe8b7c2467734848bfd42bead244a/content/common/gpu/media/gpu_video_decode_accelerator.h
[modify] https://crrev.com/bd8763305febe8b7c2467734848bfd42bead244a/content/common/gpu/media/vaapi_video_decode_accelerator.cc
[modify] https://crrev.com/bd8763305febe8b7c2467734848bfd42bead244a/content/common/gpu/media/vaapi_video_decode_accelerator.h
[modify] https://crrev.com/bd8763305febe8b7c2467734848bfd42bead244a/content/common/gpu/media/video_decode_accelerator_unittest.cc
[modify] https://crrev.com/bd8763305febe8b7c2467734848bfd42bead244a/content/common/gpu/media/vt_video_decode_accelerator_mac.cc
[modify] https://crrev.com/bd8763305febe8b7c2467734848bfd42bead244a/content/common/gpu/media/vt_video_decode_accelerator_mac.h
[modify] https://crrev.com/bd8763305febe8b7c2467734848bfd42bead244a/media/video/video_decode_accelerator.h

The following crash looks promising:
https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Mac%27%20AND%20product.version%3D%2751.0.2692.0%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27gpu-process%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BGPU%20hang%5D%20gfx%3A%3AGLContextCGL%3A%3ADestroy%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D&stbtiq=&reportid=810faaf800000000&index=12#3

We see that one thread has the stack:
  (AppleIntelHD3000GraphicsGLDriver + 0x002c538b)   
  (AppleIntelHD3000GraphicsGLDriver + 0x002c21ab)   
  (AppleIntelHD3000GraphicsGLDriver + 0x002af8f1)   
  (AppleIntelHD3000GraphicsGLDriver + 0x002f4a1b)   
  (AppleIntelHD3000GraphicsGLDriver + 0x002faeb4)   
  (libGPUSupport.dylib)     gpumDeleteCachedProgram
  (libGPUSupport.dylib)     gldDestroyContext
  (GLEngine)                gliDestroyContext
  (OpenGL)                  CGLReleaseContext
  (OpenGL)                  CGLDestroyContext
  (Google Chrome Framework) gfx::GLContextCGL::Destroy()
  (Google Chrome Framework) gfx::GLContextCGL::~GLContextCGL()
  (Google Chrome Framework) gpu::GLContextVirtual::~GLContextVirtual()
  (Google Chrome Framework) gpu::GLContextVirtual::~GLContextVirtual()
  (Google Chrome Framework) gpu::gles2::GLES2DecoderImpl::Destroy(bool)
  (Google Chrome Framework) content::GpuCommandBufferStub::Destroy()

And another thread has the stack:
  (libsystem_kernel.dylib + 0x00016162 )  __psynch_mutexwait
  (AppleGVA)                AVF_CreateMediaAcceleratorInterface
  (AppleGVA)                AVF_CreateMediaAcceleratorInterface
  (AppleGVA)                AVF_CreateMediaAcceleratorInterface
  (VideoToolbox)            VTGetGVADecoderAvailability
  (VideoToolbox)            VTDecompressionSessionDecodeFrame
  (VideoToolbox)            VTDecompressionSessionDecodeFrame
  (Google Chrome Framework) content::VTVideoDecodeAccelerator::DecodeTask(media::BitstreamBuffer const&, content::VTVideoDecodeAccelerator::Frame*)

I would hazard a guess that we're seeing some sort of deadlock between decode and CGLContext tear-down.

I'm re-introducing the patch piecemeal. It could have been decode-to-420 or draw-with-AVSampleBufferDisplayLayer that caused this. Hopefully we'll get a clear signal as to which it was.

This crash seen on latest builds as below

51.0.2693.2	2.01%	380	
51.0.2693.0	0.01%	1	
51.0.2692.0	1.30%	246	
51.0.2691.0	0.57%	107	
51.0.2690.0	0.65%	123	
51.0.2689.0	0.74%	139	
51.0.2688.0	0.02%	3	
51.0.2687.0	0.01%	2	
51.0.2686.0	0.01%	1	
51.0.2685.0	0.01%	1	
51.0.2680.0	0.01%	1	
51.0.2678.0	0.01%	1	
51.0.2673.0	0.02%	4	
51.0.2672.0	0.01%	2	
51.0.2669.0	0.01%	1	
51.0.2668.0	0.01%	1	
51.0.2667.0	0.01%	1	
51.0.2665.0	0.01%	1	
50.0.2661.57	0.01%	1	
50.0.2661.49	0.03%	6	
50.0.2661.37	0.06%	11	
50.0.2661.26	0.02%	4	
50.0.2661.18	0.01%	2	
50.0.2661.11	0.01%	1	
50.0.2659.0	0.01%	2	
50.0.2658.0	0.01%	1	
50.0.2657.0	0.02%	4	
50.0.2656.0	0.01%	1	

Link to the builds:
	
https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Mac%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27gpu-process%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BGPU%20hang%5D%20gfx%3A%3AGLContextCGL%3A%3ADestroy%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#-samplereports:5,productversion:1000

Issue 600524 has been merged into this issue.

This disappeared in 51.0.2700.0 and re-appeared in 51.0.2700.1. The difference is https://codereview.chromium.org/1851293004/, which moves us to 4:2:0 format.

So ... at least we know exactly what is causing the problem. Now we need to figure out how to fix this.

I have a guess about this: We leave GL texture bound to the Y and UV IOSurfaces while the GLImage is still around. I'll change this to delete the textures immediately and see if that fixes the problem.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f838f5f4880b9f4fe54e6e10812708c1c89b23d7

commit f838f5f4880b9f4fe54e6e10812708c1c89b23d7
Author: ccameron <ccameron@chromium.org>
Date: Thu Apr 07 07:44:35 2016

Mac h264: Work around CGLContext+VTDecompresionSession deadlock

When we converted h264 to be decoded as 4:2:0 instead of 4:2:2, we
noticed a spike in GPU crash rates, in particular, in CGLDestroyContext.
Many of these crashes had other threads in VideoToolbox in what appears
to be a deadlock.

Part of the change from 4:2:2 to 4:2:0 is adding a path where we will
do manual YUV->RGB conversion if we cannot display directly with
CoreAnimation. In the process of this conversion, we bind the planes
of the IOSurface allocated by VideoToolbox to OpenGL textures. We leave
these bound until the GLImage is destroyed.

My guess here is that the destruction of these resources in the driver
is being deferred until CGLDestroyContext (we have seen that IOSurfaces
can be kept around in the OpenGL driver long after they are unbound
in  crbug.com/158469 ), whereupon they are triggering a deadlock with
VideoToolbox, which may be re-using them.

This patch forces us to destroy the OpenGL texture objects that
reference the IOSurfaces immediately after use (and while their owning
CVPixelBuffer is still retained), which will hopefully avoid the
conflict with VideoToolbox.

Note that this is a speculative fix.

BUG= 598388 

Review URL: https://codereview.chromium.org/1862183003

Cr-Commit-Position: refs/heads/master@{#385690}

[modify] https://crrev.com/f838f5f4880b9f4fe54e6e10812708c1c89b23d7/ui/OWNERS
[modify] https://crrev.com/f838f5f4880b9f4fe54e6e10812708c1c89b23d7/ui/gl/gl_image_io_surface.h
[modify] https://crrev.com/f838f5f4880b9f4fe54e6e10812708c1c89b23d7/ui/gl/gl_image_io_surface.mm

This didn't fix the issue:  51 .0.2703.0 (cut at r385938, after the above patch) still has this as the #1 GPU process crasher.

Other ideas... issue 599314 is leaving the IOSurface bound to some texture units, so that may be some of the issue. Also, r386027 has us keep the CVPixelBufferRef retained, so maybe that is part of the issue.

It would be nice to have any idea what causes this to happen. Is this happening at GPU process shutdown?

Omg... I think I found the issue -- 99% of the calls to GLImageIOSurface::Destroy are *leaking* their GL programs.

That's why all of these stacks are hanging in gpuiCleanContextFromProgram.

So, we really need to stop creating a new GL shader for every video frame, especially if we're going to leak them...

The reason that has_context is false is that we're hitting this through
VTVideoDecodeAccelerator::ReusePictureBuffer, which releases the GLImage without a context attached.

VTVideoDecodeAccelerator will necessarily create a new GLImageIOSurface every video frame (because its IOSurfaces come from the VTDecompressionSession, and we don't have control of the reuse pool there).

I have a definitive fix at https://codereview.chromium.org/1870323002/.

Crash instances are still observed on Google Chrome Latest Stable (49.0.2623.112), Beta (50.0.2661.66), Dev (51.0.2700.0) & Canary (51.0.2704.2) channels.

Link - https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Mac%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27gpu-process%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BGPU%20hang%5D%20gfx%3A%3AGLContextCGL%3A%3ADestroy%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#-samplereports:5,productversion:1000

Users experienced this crash on the following builds:

Win Dev 51.0.2700.0 -  0.63 CPM, 486 reports, 448 clients (signature [GPU hang] content::DXVAVideoDecodeAccelerator::SendMFTMessage)
Mac Canary 52.0.2705.0 -  1.70 CPM, 4 reports, 4 clients (signature [GPU hang] gfx::GLContextCGL::Destroy)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c9295d4075c1e4a32977c446ab9734a88abf8391

commit c9295d4075c1e4a32977c446ab9734a88abf8391
Author: ccameron <ccameron@chromium.org>
Date: Tue Apr 12 19:29:27 2016

Mac h264: Plug GL program leak

The state for YUV to RGB conversion for GLImageIOSurface is stored
in the GLImageIOSurface object. This is objectionable because
- This means every GLImage object will re-create all shaders
- For hardware decode, we create a new GLImage for every decoded frame,
  which means allocating a shader for every frame.
- It is not clear which CGLContextObj the GL objects are associated
  with.
- For hardware decode, we do not specify that a GL context is current
  when we free GLImages, so we end up leaking one GL program every
  frame.
- This memory leak ends up causing GL context destroy to take more than
  ten seconds, causing a GPU hang when the GL context is destroyed.

Move the program object into an RGBConverter class. Keep a global
registry of RGBConverter classes for each CGLContextObj, and share
RGBConverter objects across GLImages. Make the RGBConverter always
destroy its objects when it is destroyed. It can do this because it
retains the CGLContextObj that was used to allocate them.

This could be better in that a pathological user of GLImages could
still cause the RGBConverter to be created and destroyed every frame.

BUG= 598388 ,599314

Review URL: https://codereview.chromium.org/1870323002

Cr-Commit-Position: refs/heads/master@{#386757}

[modify] https://crrev.com/c9295d4075c1e4a32977c446ab9734a88abf8391/ui/gl/gl_image_io_surface.h
[modify] https://crrev.com/c9295d4075c1e4a32977c446ab9734a88abf8391/ui/gl/gl_image_io_surface.mm

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c5d9e014c2adc935365cb443444c7b28e52f9ad0

commit c5d9e014c2adc935365cb443444c7b28e52f9ad0
Author: Christopher Cameron <ccameron@chromium.org>
Date: Tue Apr 12 20:50:17 2016

Mac h264: Plug GL program leak

The state for YUV to RGB conversion for GLImageIOSurface is stored
in the GLImageIOSurface object. This is objectionable because
- This means every GLImage object will re-create all shaders
- For hardware decode, we create a new GLImage for every decoded frame,
  which means allocating a shader for every frame.
- It is not clear which CGLContextObj the GL objects are associated
  with.
- For hardware decode, we do not specify that a GL context is current
  when we free GLImages, so we end up leaking one GL program every
  frame.
- This memory leak ends up causing GL context destroy to take more than
  ten seconds, causing a GPU hang when the GL context is destroyed.

Move the program object into an RGBConverter class. Keep a global
registry of RGBConverter classes for each CGLContextObj, and share
RGBConverter objects across GLImages. Make the RGBConverter always
destroy its objects when it is destroyed. It can do this because it
retains the CGLContextObj that was used to allocate them.

This could be better in that a pathological user of GLImages could
still cause the RGBConverter to be created and destroyed every frame.

BUG= 598388 ,599314

Review URL: https://codereview.chromium.org/1870323002

Cr-Commit-Position: refs/heads/master@{#386757}
(cherry picked from commit c9295d4075c1e4a32977c446ab9734a88abf8391)

Review URL: https://codereview.chromium.org/1884763003 .

Cr-Commit-Position: refs/branch-heads/2704@{#15}
Cr-Branched-From: 6e53600def8f60d8c632fadc70d7c1939ccea347-refs/heads/master@{#386251}

[modify] https://crrev.com/c5d9e014c2adc935365cb443444c7b28e52f9ad0/ui/gl/gl_image_io_surface.h
[modify] https://crrev.com/c5d9e014c2adc935365cb443444c7b28e52f9ad0/ui/gl/gl_image_io_surface.mm

Closing