Users experienced this crash on the following builds:

Win Dev 51.0.2687.0 - 125 reports, 119 clients (signature [GPU hang] igd10iumd64.dll)
Win Canary 51.0.2689.0 - 6 reports, 6 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Canary 52.0.2707.0 -  2.08 CPM, 7 reports, 5 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Canary 53.0.2748.0 -  0.65 CPM, 8 reports, 5 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Moving this nonessential bug to the next milestone.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Users experienced this crash on the following builds:

Win Beta 51.0.2704.63 -  2.65 CPM, 3232 reports, 2644 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Beta 52.0.2743.24 -  0.49 CPM, 27 reports, 24 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Beta 52.0.2743.24 -  0.49 CPM, 27 reports, 24 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Beta 52.0.2743.24 -  0.53 CPM, 245 reports, 212 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Canary 53.0.2760.0 -  0.52 CPM, 23 reports, 21 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Stable 51.0.2704.106 -  0.48 CPM, 643 reports, 557 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Canary 54.0.2786.0 -  0.63 CPM, 8 reports, 6 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

This issue is Pri-1 but has already been moved once. Lowering the priority and moving to the next milestone.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Users experienced this crash on the following builds:

Win Canary 55.0.2841.0 -  0.66 CPM, 7 reports, 6 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Canary 56.0.2884.0 -  1.91 CPM, 11 reports, 8 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Canary 57.0.2925.0 -  0.57 CPM, 6 reports, 6 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Canary 58.0.2988.0 -  0.65 CPM, 8 reports, 8 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Canary 59.0.3030.0 -  0.44 CPM, 6 reports, 6 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Issue 699642 has been merged into this issue.

I don't think there's anything for stability sheriff to do here.

Here's a representative sample of the gpu hangs for the most recent one

https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27gpu-process%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%20LIKE%20%27%5BGPU%20hang%5D%25igd10iumd64%25%27%20AND%20product.Version%3D%2759.0.3030.0%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D

=> a smattering of Intel hardware on a smattering of drivers.

Do we attempt to do anything about GPU hangs?

We don't normally do much about GPU hangs unless there's a spike in frequency or they're really specific to a driver version we can blacklist.

Users experienced this crash on the following builds:

Win Canary 60.0.3074.0 -  0.40 CPM, 19 reports, 19 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Just to update, latest crash rates on all channels are as below:
This crash not seen on latest Canary.

60.0.3089.0	0.00%	10	
60.0.3088.3	0.00%	11	-Dev
59.0.3071.36	0.02%	96	-Beta
58.0.3029.96	1.44%	8792	-Stable

Link to the list of builds:
---------------------------
https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.ptype%3D%27gpu-process%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BGPU%20hang%5D%20igd10iumd64.dll%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#samplereports:5,productversion:1000,+magicsignature

Seeing huge spikes in latest stable some one from cc please look in to this issue.

Thank You!

Users experienced this crash on the following builds:

Win Canary 61.0.3117.0 -  0.50 CPM, 8 reports, 8 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

I did a version comparison for Current vs Previous Beta (60.0.3112.40 vs 60.0.3112.50) and seeing a little spike in Latest Beta# 60.0.3112.50 w/ 916 crash instances.

Version Comparison: https://crash.corp.google.com/browse?q=product.name%3D%27Chrome%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BGPU%20hang%5D%20igd10iumd64.dll%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D&compProp=product.Version&v1=60.0.3112.40&v2=60.0.3112.50

Could someone please see if there is a related change in the below change log?
https://chromium.googlesource.com/chromium/src/+log/60.0.3112.40..60.0.3112.50?pretty=fuller&n=10000

Thank you!

Just to update the latest behaviour of crash,

This crash is seen in all channels as below

61.0.3153.0	0.01%	90        --- Latest Dev
60.0.3112.66	0.01%	115       --- Latest Beta
59.0.3071.115	13.75%	168236    --- Latest Stable	
Crash is not seen in latest canary.

Link to the builds:
https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.ptype%3D%27gpu-process%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BGPU%20hang%5D%20igd10iumd64.dll%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#samplereports:5,productversion:1000,+magicsignature

Could any dev from Internals>GPU team please have a look into the issue as it has been marked as a stable blocker.

Thanks...!!

Latest update of the crash behavior as below

61.0.3161.0	0.00%	4	
61.0.3159.5	0.00%	29	
60.0.3112.72	0.00%	10	
59.0.3071.115	18.51%	241064	

Link to the list of builds
https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.ptype%3D%27gpu-process%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BGPU%20hang%5D%20igd10iumd64.dll%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#samplereports:5,productversion:1000,+magicsignature

Could some once please look into this issue as this is marked as stable blocker for M-60.

Thanks,

Just to update the latest behaviour of crash,

This crash is seen in all channels as below

62.0.3164.0	0.00%	31        --- Latest Canary
61.0.3159.5	0.01%	166       --- Latest Dev
60.0.3112.72	0.04%	518       --- Latest Beta
59.0.3071.115	21.57%	292933    --- Latest Stable

Seeing huge crashes on all the channels as above.	
Could any dev from Internals>GPU team please have a look into the issue as it has been marked as a stable blocker.

Thanks...!!

Bumping to 61 as we're very close to M60 stable and only taking critical bug fixes.

URGENT - PTAL.
Your bug is labelled as Stable ReleaseBlock, pls make sure to land the fix and get it merged into the M61 branch #3163 ASAP to have enough baking time in Beta before Stable promotion. Thank you!

Know that this issue shouldn't block the release?  Remove the ReleaseBlock-Stable label.

I don't see any regression in M60. The analysis in comment 27 is flawed because it compares against 60.0.3112.40, which had a bug that caused the GPU process to crash before it got to this code.

Users experienced this crash on the following builds:

Win Dev 63.0.3218.0 -  0.71 CPM, 47 reports, 45 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Beta 62.0.3202.29 -  1.00 CPM, 997 reports, 805 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Canary 64.0.3243.0 -  1.87 CPM, 22 reports, 20 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Just to update the latest behavior, Still crash instances are observed across all channels, currently this crash is ranked as number #7 under gpu process for Windows platform. Below information provides the comparison between previous and latest channels including total number of instances.

+--------------------------------------------------+      
| Latest Channel        |    Previous Channel      |
|--------------------------------------------------+
|62.0.3202.94   - 286920|   62.0.3202.89  - 75744  |--> Stable
|63.0.3239.59   - 64    |   63.0.3239.52  - 2696   |--> Beta
|64.0.3269.3    - 786   |   64.0.3260.2   - 1615   |--> Dev
|64.0.3274.0    - 0     |   64.0.3273.3   -  87    |--> Canary
+--------------------------------------------------+
	
Link to the list of the builds getting crash:
---------------------------------------------
https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.ptype%3D%27gpu-process%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BGPU%20hang%5D%20igd10iumd64.dll%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#-property-selector,-samplereports:5,productversion:1000,+magicsignature

Thanks!	

Just to update:

[GPU hang] igd10iumd64.dll

65.0.3291.0	0.00%	65	-Canary
65.0.3290.2	0.00%	18	

This is top#1 GPU process crash on latest Canary-65.0.3291.0 & still seeing 50 instances from 41 clients so far on Windows.

Link to the list of builds:
-------------------------
https://crash.corp.google.com/browse?q=custom_data.ChromeCrashProto.ptype%3D%27gpu-process%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BGPU%20hang%5D%20igd10iumd64.dll%27&sql_dialect=googlesql&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D

Could some one from cc'ed dev, please take a look.
Thanks..!

This is 97.7% Intel.

Yang, do you have any insights on this?

1	0x8086 0x1916	16.12%	604208	Sky Lake Integrated Graphics	
2	0x8086 0x0a16	11.21%	420480	Haswell-ULT Integrated Graphics Controller	
3	0x8086 0x5916	10.78%	404060		
4	0x8086 0x0f31	10.51%	393935	Atom Processor Z36xxx/Z37xxx Series Graphics & Display	
5	0x8086 0x0166	9.31%	348999	3rd Gen Core processor Graphics Controller	
6	0x8086 0x1912	7.37%	276199	Sky Lake Integrated Graphics	
7	0x8086 0x1616	5.20%	194840	Broadwell-U Integrated Graphics	
8	0x8086 0x0156	3.77%	141427	3rd Gen Core processor Graphics Controller	
9	0x8086 0x0416	3.32%	124363	4th Gen Core Processor Integrated Graphics Controller	
10	0x8086 0x191b

This file is part of Intel GPU driver. What does 97.7% Intel mean?
I checked internal bug tracking system of GPU driver, and found some recent crash issues about this DLL. But without more detailed info, I couldn't figure how relevant they are. Do you have version of this file in crash report? It would be very helpful to track the behavior of driver issues.

97.7% is among all crashes that has this stack signature, 97.7% are with Intel GPUs.

We don't have the version number of this DLL, but we do have the version number of drivers.

1	22.20.16.4749	15.16%	
2	22.20.16.4771	6.67%
3	10.18.10.4276	4.75%
4	22.20.16.4691	3.88%
5	21.20.16.4727	3.51%
6	20.19.15.4531	3.13%
7	10.18.10.3496	3.08%
8	22.20.16.4799	2.87%
9	10.18.10.4358	2.86%
10	10.18.10.3958	2.14%
11	22.20.16.4708	2.11%
12	10.18.15.4248	2.06%
13	10.18.10.3412	1.99%
14	10.18.10.3408	1.80%
15	10.18.10.3621	1.76%
16	10.18.10.3308	1.59%
17	10.18.14.4264	1.58%
18	20.19.15.4549	1.49%
19	10.18.10.4425	1.42%
20	10.18.10.3368	1.01%

The driver version is good enough. Version 22.20.16.4749 was released on July 24, and my current desktop (Windows 10 Fall Creators Update) is using exactly the same driver. However, today it asked me to upgrade to 22.20.16.4836 that is released on Oct 17. Maybe we can continue to observe if there are new issues coming from version 4836. 
Could you please paste the crash stack here if it's different from #1? I cross-checked all 9 internal issues with this DLL, but couldn't find a match like the stack in #1. 
And if you think this is a high priority issue, I may also file an issue in internal bug tracking system.

Yang, this is top #1 crash on Canary, so I think it's worth looking into what's going on.

@zmo, do you have a new crash stack for this issue? The stack listed in comment #1 was posted more than 1.5 years ago, and I don't have access right to link listed in comment #38. 

Yang, below is from a crash today. Note that Thread 0 is GPU main thread, which is what hangs. Thread 1 is watch dog thread, which after waiting a few minutes, kill the GPU process. From Thread 2 to 21, they are all the same, and seem to be Driver's internal threads.

Total Threads22
Processed Threads22

Thread 0 (id: 3608) MAGIC SIGNATURE THREAD
Stack Quality40%Show frame trust levels
0x00007ffcb1664324	(win32u.dll + 0x00004324 )	NtGdiDdDDIDestroyAllocation
0x00007ffcae3313ad	(d3d11.dll + 0x000d13ad )	NDXGI::CDevice::DeallocateCB(void *,_D3DDDICB_DEALLOCATE const *)
0x00007ffcabdfd4ce	(igd10iumd64.dll + 0x0002d4ce )	
0x00007ffcabdfd549	(igd10iumd64.dll + 0x0002d549 )	
0x00007ffcabe14320	(igd10iumd64.dll + 0x00044320 )	

Thread 1 (id: 9584) CRASHED [EXCEPTION_ACCESS_VIOLATION_WRITE @ 0x00000000 ]
Stack Quality100%Show frame trust levels
0x00007ffc7ec17f30	(chrome_child.dll -gpu_watchdog_thread.cc:421 )	gpu::GpuWatchdogThread::DeliberatelyTerminateToRecoverFromHang()
0x00007ffc7ec180e7	(chrome_child.dll -gpu_watchdog_thread.cc:291 )	gpu::GpuWatchdogThread::OnCheckTimeout()
0x00007ffc7d2813c1	(chrome_child.dll -bind_internal.h:318 )	base::internal::Invoker<base::internal::BindState<void ( cc::ResourcePool::*)(void),base::WeakPtr<cc::ResourcePool> >,void >::RunOnce(base::internal::BindStateBase *)
0x00007ffc7d30eb8d	(chrome_child.dll -task_annotator.cc:57 )	base::debug::TaskAnnotator::RunTask(char const *,base::PendingTask *)
0x00007ffc7d29edd0	(chrome_child.dll -message_loop.cc:394 )	base::MessageLoop::RunTask(base::PendingTask *)
0x00007ffc7d3b9e3a	(chrome_child.dll -message_loop.cc:406 )	base::MessageLoop::DeferOrRunPendingTask(base::PendingTask)
0x00007ffc7d2a2047	(chrome_child.dll -message_loop.cc:490 )	base::MessageLoop::DoDelayedWork(base::TimeTicks *)
0x00007ffc7d30e052	(chrome_child.dll -message_pump_default.cc:41 )	base::MessagePumpDefault::Run(base::MessagePump::Delegate *)
0x00007ffc7d41f55b	(chrome_child.dll -message_loop.cc:345 )	base::MessageLoop::Run(bool)
0x00007ffc7d41f4fb	(chrome_child.dll -run_loop.cc:114 )	base::RunLoop::Run()
0x00007ffc7d41f3f9	(chrome_child.dll -thread.cc:338 )	base::Thread::ThreadMain()
0x00007ffc7d41df9f	(chrome_child.dll -platform_thread_win.cc:89 )	base::`anonymous namespace'::ThreadFunc
0x00007ffcb2b82773	(KERNEL32.DLL + 0x00012773 )	BaseThreadInitThunk
0x00007ffcb4c30d60	(ntdll.dll + 0x00070d60 )	RtlUserThreadStart

Thread 2 (id: 8632)
Stack Quality100%Show frame trust levels
0x00007ffcb4c65434	(ntdll.dll + 0x000a5434 )	NtWaitForSingleObject
0x00007ffcb127988e	(KERNELBASE.dll + 0x0003988e )	WaitForSingleObjectEx
0x00007ffc7d2c4975	(chrome_child.dll -waitable_event_win.cc:61 )	base::WaitableEvent::Wait()
0x00007ffc7d30e0e9	(chrome_child.dll -message_pump_default.cc:57 )	base::MessagePumpDefault::Run(base::MessagePump::Delegate *)
0x00007ffc7d41f55b	(chrome_child.dll -message_loop.cc:345 )	base::MessageLoop::Run(bool)
0x00007ffc7d41f4fb	(chrome_child.dll -run_loop.cc:114 )	base::RunLoop::Run()
0x00007ffc7d41f3f9	(chrome_child.dll -thread.cc:338 )	base::Thread::ThreadMain()
0x00007ffc7d41df9f	(chrome_child.dll -platform_thread_win.cc:89 )	base::`anonymous namespace'::ThreadFunc
0x00007ffcb2b82773	(KERNEL32.DLL + 0x00012773 )	BaseThreadInitThunk
0x00007ffcb4c30d60	(ntdll.dll + 0x00070d60 )	RtlUserThreadStart

Thread 3 (id: 9028)
Stack Quality100%Show frame trust levels
0x00007ffcb4c65434	(ntdll.dll + 0x000a5434 )	NtWaitForSingleObject
0x00007ffcb127988e	(KERNELBASE.dll + 0x0003988e )	WaitForSingleObjectEx
0x00007ffc7d3b8fd6	(chrome_child.dll -waitable_event_win.cc:84 )	base::`anonymous namespace'::WaitUntil
0x00007ffc7d3ba889	(chrome_child.dll -waitable_event_win.cc:116 )	base::WaitableEvent::TimedWait(base::TimeDelta const &)
0x00007ffc7d3baea9	(chrome_child.dll -scheduler_worker.cc:171 )	base::internal::SchedulerWorker::Delegate::WaitForWork(base::WaitableEvent *)
0x00007ffc7d3baca9	(chrome_child.dll -scheduler_worker.cc:68 )	base::internal::SchedulerWorker::Thread::ThreadMain()
0x00007ffc7d41df9f	(chrome_child.dll -platform_thread_win.cc:89 )	base::`anonymous namespace'::ThreadFunc
0x00007ffcb2b82773	(KERNEL32.DLL + 0x00012773 )	BaseThreadInitThunk
0x00007ffcb4c30d60	(ntdll.dll + 0x00070d60 )	RtlUserThreadStart

Here is another one from today (better stack quality):

Thread 0 (id: 1672) MAGIC SIGNATURE THREAD
Stack Quality95%Show frame trust levels
0x00007ff88a935a24	(ntdll.dll + 0x000a5a24 )	NtDelayExecution
0x00007ff886f37216	(KERNELBASE.dll + 0x00057216 )	SleepEx
0x00007ff875a52f2e	(igd10iumd64.dll + 0x00592f2e )	
0x00007ff85d87af1a	(libglesv2.dll -statemanager11.cpp:167 )	rx::`anonymous namespace'::SortAttributesByLayout
0x00007ff85d87fac1	(libglesv2.dll -statemanager11.cpp:2573 )	rx::StateManager11::applyVertexBuffer(gl::Context const *,unsigned int,int,int,int,rx::TranslatedIndexData *)
0x00007ff85d7efc5e	(libglesv2.dll -renderer11.cpp:1713 )	rx::Renderer11::drawElements(gl::Context const *,unsigned int,int,unsigned int,void const *,int)
0x00007ff85d86a6e6	(libglesv2.dll -context11.cpp:183 )	rx::Context11::drawElements(gl::Context const *,unsigned int,int,unsigned int,void const *)
0x00007ff85d776ecd	(libglesv2.dll -context.cpp:1811 )	gl::Context::drawElements(unsigned int,int,unsigned int,void const *)
0x00007ff85d7527ae	(libglesv2.dll -entry_points_gles_2_0_autogen.cpp:767 )	gl::DrawElements(unsigned int,int,unsigned int,void const *)
0x00007ff84cee6829	(chrome_child.dll -gl_gl_api_implementation.cc:474 )	gl::RealGLApi::glDrawElementsFn(unsigned int,int,unsigned int,void const *)
0x00007ff84d81f4c1	(chrome_child.dll -gles2_cmd_decoder.cc:10533 )	gpu::gles2::GLES2DecoderImpl::DoDrawElements(char const *,bool,unsigned int,int,unsigned int,int,int)
0x00007ff84d82a23a	(chrome_child.dll -gles2_cmd_decoder.cc:10569 )	gpu::gles2::GLES2DecoderImpl::HandleDrawElements(unsigned int,void const volatile *)
0x00007ff84b7f1cdd	(chrome_child.dll -gles2_cmd_decoder.cc:5373 )	gpu::gles2::GLES2DecoderImpl::DoCommandsImpl<0>(unsigned int,void const volatile *,int,int *)
0x00007ff84b7f1a0f	(chrome_child.dll -gles2_cmd_decoder.cc:5424 )	gpu::gles2::GLES2DecoderImpl::DoCommands(unsigned int,void const volatile *,int,int *)
0x00007ff84b7f1966	(chrome_child.dll -command_buffer_service.cc:90 )	gpu::CommandBufferService::Flush(int,gpu::AsyncAPIInterface *)
0x00007ff84b620579	(chrome_child.dll -gpu_command_buffer_stub.cc:1010 )	gpu::GpuCommandBufferStub::OnAsyncFlush(int,unsigned int,std::vector<ui::LatencyInfo,std::allocator<ui::LatencyInfo> > const &)
0x00007ff84b6213da	(chrome_child.dll -ipc_message_templates.h:146 )	IPC::MessageT<GpuCommandBufferMsg_AsyncFlush_Meta,std::tuple<int,unsigned int,std::vector<ui::LatencyInfo,std::allocator<ui::LatencyInfo> > >,void>::Dispatch<gpu::GpuCommandBufferStub,gpu::GpuCommandBufferStub,void,void ( gpu::GpuCommandBufferStub::*)(int,unsigned int,std::vector<ui::LatencyInfo,std::allocator<ui::LatencyInfo> > const &)>(IPC::Message const *,gpu::GpuCommandBufferStub *,gpu::GpuCommandBufferStub *,void *,void ( gpu::GpuCommandBufferStub::*)(int,unsigned int,std::vector<ui::LatencyInfo,std::allocator<ui::LatencyInfo> > const &))
0x00007ff84baf50d1	(chrome_child.dll -gpu_command_buffer_stub.cc:308 )	gpu::GpuCommandBufferStub::OnMessageReceived(IPC::Message const &)
0x00007ff84b7ee7aa	(chrome_child.dll -gpu_channel.cc:1037 )	gpu::GpuChannel::HandleMessageHelper(IPC::Message const &)
0x00007ff84b7ee6ea	(chrome_child.dll -gpu_channel.cc:985 )	gpu::GpuChannel::HandleMessage(IPC::Message const &)
0x00007ff84b7ee691	(chrome_child.dll -bind_internal.h:331 )	base::internal::Invoker<base::internal::BindState<void ( gpu::GpuChannel::*)(IPC::Message const &),base::WeakPtr<gpu::GpuChannel>,IPC::MessageT<GpuCommandBufferMsg_AsyncFlush_Meta,std::tuple<int,unsigned int,std::vector<ui::LatencyInfo,std::allocator<ui::LatencyInfo> > >,void> >,void >::RunOnce(base::internal::BindStateBase *)
0x00007ff84ba0da66	(chrome_child.dll -callback.h:64 )	base::OnceCallback<void >::Run( ?? )
0x00007ff84b7f07f6	(chrome_child.dll -scheduler.cc:546 )	gpu::Scheduler::RunNextTask()
0x00007ff84b66a40f	(chrome_child.dll -bind_internal.h:331 )	base::internal::Invoker<base::internal::BindState<void ( media::MultibufferDataSource::*)(void),base::WeakPtr<media::MultibufferDataSource> >,void >::Run(base::internal::BindStateBase *)
0x00007ff84b64eb8d	(chrome_child.dll -task_annotator.cc:57 )	base::debug::TaskAnnotator::RunTask(char const *,base::PendingTask *)
0x00007ff84b5dedd0	(chrome_child.dll -message_loop.cc:394 )	base::MessageLoop::RunTask(base::PendingTask *)
0x00007ff84b64f051	(chrome_child.dll -message_loop.cc:450 )	base::MessageLoop::DoWork()
0x00007ff84b64e034	(chrome_child.dll -message_pump_default.cc:37 )	base::MessagePumpDefault::Run(base::MessagePump::Delegate *)
0x00007ff84b75f55b	(chrome_child.dll -message_loop.cc:345 )	base::MessageLoop::Run(bool)
0x00007ff84b75f4fb	(chrome_child.dll -run_loop.cc:114 )	base::RunLoop::Run()
0x00007ff84b81e6a3	(chrome_child.dll -gpu_main.cc:318 )	content::GpuMain(content::MainFunctionParams const &)
0x00007ff84b94f611	(chrome_child.dll -content_main_runner.cc:430 )	content::RunNamedProcessTypeMain(std::basic_string<char,std::char_traits<char>,std::allocator<char> > const &,content::MainFunctionParams const &,content::ContentMainDelegate *)
0x00007ff84b823b7b	(chrome_child.dll -content_main_runner.cc:710 )	content::ContentMainRunnerImpl::Run()
0x00007ff84b95018e	(chrome_child.dll -main.cc:469 )	service_manager::Main(service_manager::MainParams const &)
0x00007ff84b605b86	(chrome_child.dll -content_main.cc:19 )	content::ContentMain(content::ContentMainParams const &)
0x00007ff84b605b29	(chrome_child.dll -chrome_main.cc:123 )	ChromeMain
0x00007ff78f757163	(chrome.exe -main_dll_loader_win.cc:199 )	MainDllLoader::Launch(HINSTANCE__ *,base::TimeTicks)
0x00007ff78f75256d	(chrome.exe -chrome_exe_main_win.cc:230 )	wWinMain
0x00007ff78f8273e2	(chrome.exe -exe_common.inl:283 )	__scrt_common_main_seh
0x00007ff889502773	(KERNEL32.DLL + 0x00012773 )	BaseThreadInitThunk
0x00007ff88a900d50	(ntdll.dll + 0x00070d50 )	RtlUserThreadStart

The crash stacks are very helpful! I have filed an internal issue, and will bring back any feedback from our Windows graphics driver team. 

@zmo, our Windows graphics driver team asked for app crash dump file. If you have, could you please provide it so that I can upload to internal system? Thanks!

Sent through email. Let me assign this to you, waiting for Intel driver team feedback.

I just got below analysis from our graphics team (some internal info are removed): 

I analyzed the memory dump from ([Yang] Internal link is removed) which was delivered by zmo user from chromium community - Thread 0 (Imola UMD driver) has got hung because it is waiting for the thread 18 from the IGC thread pool to finish pixel shader compilation. 

Having a look into comment listing most affected platforms  https://bugs.chromium.org/p/chromium/issues/detail?id=597700#c39 :
This is 97.7% Intel. Yang, do you have any insights on this? 1 0x8086 0x1916 16.12% 604208 Sky Lake Integrated Graphics 2 0x8086 0x0a16 11.21% 420480 Haswell-ULT Integrated Graphics Controller 3 0x8086 0x5916 10.78% 404060 4 0x8086 0x0f31 10.51% 393935 Atom Processor Z36xxx/Z37xxx Series Graphics & Display 5 0x8086 0x0166 9.31% 348999 3rd Gen Core processor Graphics Controller 6 0x8086 0x1912 7.37% 276199 Sky Lake Integrated Graphics 7 0x8086 0x1616 5.20% 194840 Broadwell-U Integrated Graphics 8 0x8086 0x0156 3.77% 141427 3rd Gen Core processor Graphics Controller 9 0x8086 0x0416 3.32% 124363 4th Gen Core Processor Integrated Graphics Controller 10 0x8086 0x191b
We see mostly the weak CPU platforms - ULT, atom. So I just suspect that the problem occurs when system is under a heavy load (CPU to be more specific)

The question is -  why is the compilation taking so long - only because of pure CPU performance or can we optimize sth? And how long is it taking exactly?

Can we get more info on the length of the timeout? User zmo@ https://bugs.chromium.org/p/chromium/issues/detail?id=597700#c45  mentioned ''after waiting a few minutes" Can we ask for more details on how long the timeout is? And another question - is the chromium community able to deliver this particular pixel shader that is being compiled during the hang? Maybe a full process memory dump so we could dig out the shader out of it?

Our GPU watchdog timeout is set to 15s on Windows.

No, we are not able to get the pixel shader out of crash reports, unfortunately.

Add a few ANGLE folks.

I vaguely remember we did some tweaking on D3D shader compilation before, but forgot the details.

What we might be able to do is, at time when watchdog kills the GPU process, we take a look at the context that's hanging. If it's internal contexts, for example, compositor, then we might be able to log a field indicating which internal shader compilation is hanging.

I think there is very few possibility that timeout is triggered by shader inside Chrome. I'm not sure if there are other effective ways to encourage the user to report this issue, so that we can have more info, such as the web page they visit. 
BTW, some internal discussions are still arising to alleviate this situation, and I will bring them back once we have some conclusions. 

Users experienced this crash on the following builds:

Win Canary 65.0.3323.0 -  3.04 CPM, 62 reports, 59 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Canary 66.0.3335.0 -  3.01 CPM, 54 reports, 49 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

This crash has high impact on Chrome's stability.
Signature: [GPU hang] igd10iumd64.dll.
Channel: canary. Platform: win.
Labeling issue 597700 with ReleaseBlock-Dev.


If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Remove Release-block because this is not a Chrome regression.

Intel might have a fix on their side for these crashes, but they want to find a way to test it before releasing to the public.

 Issue 811849  has been merged into this issue.

Have there been any updates on this bug?

Users experienced this crash on the following builds:

Win Beta 65.0.3325.125 -  6.52 CPM, 4628 reports, 3130 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Enterprise customer has reported this issue.

Windows 10 Home 
Version: 1709 OS Build: 16299.248 
Google Chrome version: 64.0.3282.167 
[GPU hang] igd10iumd64.dll
Report ID.
8ceb99b81c54fe4d
e1f9bf7453271dc9
Case #15037496

Do they have a case to repro this? 

Users experienced this crash on the following builds:

Win Dev 66.0.3359.26 -  2.79 CPM, 566 reports, 455 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Win Canary 67.0.3372.0 -  3.46 CPM, 70 reports, 63 clients (signature [GPU hang] igd10iumd64.dll)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Intel just rolled out a new driver that potentially fixes this crash

https://downloadcenter.intel.com/download/27680/Graphics-Intel-Graphics-Driver-for-Windows-10?product=80939

Let's wait and see if we see crash reports on this driver or not

Since Intel folks don't have access to Google crash database, I'll take this bug from yang.gu@intel.com

Issue 840786 has been merged into this issue.

Issue 846462 has been merged into this issue.

Do you have the driver version of Issue 846462?

The GPU info from the crash in Issue 846462 is:

device id	0x8086 0x1916	Sky Lake Integrated Graphics
vendor id	0x8086	Intel Corporation
driver	21.20.16.4821	

The stack looks similar to comment #46 but worse quality.

The driver supposed to contain the fix is >= 24.20.100.6025. The driver in this issue (Issue 846462) seems a bit older. Let's continue to monitor the crash reports from new driver. 

Issue 854138 has been merged into this issue.

Just to update the latest behavior of this issue in the latest channels:

Still seeing 4158435 crashes from 1642864 clients so far on latest Stable - 67.0.3396.99 on Windows OS. This crash is ranked as number #1 in 'GPU-process' Stable crashes. 

69.0.3493.3	0.02%	3086 - Dev
68.0.3440.68	0.06%	9477 - Beta
67.0.3396.99	26.42%	4161181 - Stable
So far no crash instances are observed on latest Canary.

Link to the list of builds:
-------------------------
https://crash.corp.google.com/browse?q=product_name%3D%27Chrome%27+AND+expanded_custom_data.ChromeCrashProto.channel%3D%27%27+AND+expanded_custom_data.ChromeCrashProto.ptype%3D%27gpu-process%27+AND+expanded_custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BGPU+hang%5D+igd10iumd64.dll%27

Thanks!

GPU DRIVER VERSION			
1	22.20.16.4749	18.44%	2333828
2	23.20.16.4973	17.76%	2247894
3	22.20.16.4836	11.47%	1451642
4	22.20.16.4771	7.17%	907481
5	22.20.16.4708	4.92%	622812
6	21.20.16.4727	3.87%	490458
7	23.20.16.4849	3.51%	443845
8	22.20.16.4691	2.96%	374041
9	22.20.16.4815	2.94%	371993
10	10.18.10.4358	2.55%	322479
11	21.20.16.4550	1.95%	246690
12	21.20.16.4821	1.92%	242850
13	20.19.15.4531	1.83%	231899
14	20.19.15.4835	1.53%	194160
15	10.18.15.4248	1.36%	172700
16	22.20.16.4799	1.36%	171791
17	20.19.15.4549	1.27%	160142
18	20.19.15.4568	0.97%	122887
19	22.20.16.4735	0.92%	116499
20	22.20.16.4718	0.81%	102332
21	21.20.16.4534	0.61%	77285
22	23.20.16.4877	0.60%	75653
23	20.19.15.4352	0.55%	69857
24	21.20.16.4627	0.53%	66679
25	10.18.10.4276	0.51%	64439
26	23.20.16.4901	0.50%	63402
27	20.19.15.4642	0.46%	58487
28	20.19.15.4703	0.39%	49162
29	20.19.15.4463	0.39%	48743
30	20.19.15.4509	0.35%	44100
31	10.18.15.4256	0.35%	43673
32	20.19.15.4300	0.32%	40193
33	20.19.15.4380	0.29%	37290
34	10.18.10.4425	0.28%	35023
35	22.20.16.4785	0.26%	32514
36	21.20.16.4590	0.23%	28554
37	21.20.16.4599	0.22%	27409
38	21.20.16.4542	0.21%	27020
39	10.18.10.4653	0.20%	25320
40	23.20.16.5018	0.16%	20708
41	21.20.16.4574	0.16%	19759
42	23.20.16.4982	0.14%	18251
43	23.20.16.4944	0.14%	17986
44	20.19.15.4364	0.13%	16772
45	20.19.15.4624	0.13%	16460
46	23.20.16.4905	0.12%	15652
47	20.19.15.4320	0.11%	14123
48	20.19.15.4474	0.09%	11692
49	21.20.16.4565	0.09%	11603
50	22.20.16.4811	0.08%	10416

Total:	98.10%	12416648

There is no crash from drivers >= 24.20.100.6025

This indicates the newer Intel drivers might fix the crash we see in this bug.

Although it's unfortunate this is still #1 GPU crash in stable channel, but the good news is it's going to become less and less as more users install the latest drivers.

Let's keep minoring the crashes.

Hat off to our Intel colleagues.

Issue 871518 has been merged into this issue.

 Issue 872708  has been merged into this issue.

After  crbug.com/838970  is fixed (driver of bots will be upgraded to 24.20.100.6286), please help to observe if there is any new crash reports coming from your bots with new driver.

We still have

24.20.100.6025 0.14%	27455
24.20.100.6194	0.04%	8831
24.20.100.6229	0.02%	3765
24.20.100.6136	0.01%	2965
24.20.100.6170	0.01%	2906
24.20.100.6168	0.01%	1686
24.20.100.6286	0.01%	1145

So yang.gu@, do you have driver adoption data on Intel side? If 24.20.100.6286 adoption is high (way higher than 0.01%), then we could consider this bug fixed. Otherwise, the low crash counts may simply because this driver isn't widely spread yet.

Issue 901006 has been merged into this issue.

Issue 903243 has been merged into this issue.

24.20.100.6344	0.46%	135958
24.20.100.6025	0.31%	97418
24.20.100.6286	0.25%	79963
24.20.100.6170	0.06%	19429
24.20.100.6136	0.05%	16988
24.20.100.6194	0.05%	16868
24.20.100.6229	0.05%	15784

Issue 791051 has been merged into this issue.