Tested this issue on Windows 10, Ubuntu 1.04 and Mac 10.12.5 with chrome #58.0.3029.96 as per steps mentioned in the comment #0.

Observed no crash on these machines.

rasha808@ Could you retry the scenario and help us with the server Id from "chrome://crashes" for further triage

Thank You...

kkaluri@chromium.org -- I just observed the "Rats! WebGL hit a snag" message on a Google Keep tab; however there is no corresponding crash in chrome://crashes - the most recent one is from yesterday: Crash ID 1d11c8e0-5b08-4382-b4da-4795764cec45

Thank you for providing more feedback. Adding requester "kkaluri@chromium.org" to the cc list and removing "Needs-Feedback" label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

kkaluri@chromium.org -- I just reproduced the "Rats!" message on a Maps tab by doing a suspend and resume: Crash ID fe140a64-8952-44d9-aa2c-1cb1dda7d59b

rasha808@ Could you help us with the server Id from "chrome://crashes" for further triage
Check the attached screenshot for reference.

Thank You..

Deleted: issue 729686.PNG 7.4 KB

	issue 729686.PNG 7.4 KB View Download

Crash ID ea894921-17c6-4760-a79b-558fd6888f36 (Server ID: 9f79226e40000000)

Crash report captured on Monday, June 12, 2017 at 5:54:42 PM, uploaded on Monday, June 12, 2017 at 7:53:10 PM

_________________________________________

Crash ID fe140a64-8952-44d9-aa2c-1cb1dda7d59b (Server ID: 764ab553f0000000)

Crash report captured on Monday, June 12, 2017 at 3:36:00 PM, uploaded on Monday, June 12, 2017 at 3:51:16 PM

_________________________________________

Crash ID 1d11c8e0-5b08-4382-b4da-4795764cec45 (Server ID: 846fe553f0000000)

Crash report captured on Sunday, June 11, 2017 at 10:54:38 PM, uploaded on Monday, June 12, 2017 at 3:51:15 PM

Is it useful for me to continue to upload additional Crash reports? Because this happens every time I resume from suspend.

Thank you for providing more feedback. Adding requester "kkaluri@chromium.org" to the cc list and removing "Needs-Feedback" label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

The stack trace of id "764ab553f0000000" is similar to issue 694540, hence merging into it.

I'm going to un-merge this and instead block it on the other bug. It looks like something's wrong with the GPU watchdog. Chrome should be watching for suspend/resume events and suspending and resuming the watchdog. The watchdog also skips firing under certain situations to avoid confusion by suspend/resume. John, looking at the stack traces here, can you think of why the heuristics might be failing? This is an Intel HD 4000, so presumably an older and slower machine.

Stanis, did you work on the GPU watchdog recently and could some of your changes have been responsible?

The last suspend and resume time should be preserved in the crash dump. That will tell us if the suspend/resume logic of the watchdog worked as expected. I'll take a look at the dump.

I looked at 3 crash dumps mentioned above in comment #8 and none of them has recorded suspend time and resume time (these are members of GpuWatchdogThread). We should investigate if this mechanism of suspend/resume notification of the watchdog works reliably. 

Furthermore here are timestamps I found in the 3rd crash dump:
Check timeticks (ms): 1,803,546,709
Crash timeticks (ms): 1,803,561,711
Task start (ms):      1,803,539,431 (this is from the main thread call stack, TaskAnnotator::RunTask frame).

The difference between the first two timestamps is 15 seconds as expected. The 3rd timestamp shows that the task in the main GPU thread started ~7 seconds before the check and ~22 seconds before the crash.

If I remember correctly power notifications come through the main GPU thread before being propagated to the watchdog thread. If the main GPU thread was already stuck, I suspect that would have prevented power notifications from being delivered.

Yeah, I thought PowerMonitorBroadcastSource was listening on the IO thread, but it looks like with mojo it's actually running on the GPU main thread.

+ke.he@intel

Yep. Looks like https://codereview.chromium.org/2433203003 binds the mojo-ipc on the main-thread, whereas the chrome-ipc message-filter used to process the IPC on the IO thread.

PowerMonitorBroadcastSource should probably know about the IO thread, and do some thread hopping to make sure it receives the mojo messages on the IO thread.

... although, it looks like in the old code, we would process the chrome ipc on the IO thread, but then hop back onto the main-thread to do any actual work?

sadrul@, Yep, The mojo-ipc is bound on main-thread.

In the old code, see "channel_->AddFilter()" in ChildThreadImpl, all the messages are handled in IO thread. That's why the PowerMonitorBroadcastSource have to receive messages first on IO thread then post tasks onto main-thread. All the actual work happens on main-thread.

Now we convert the old IPC to mojo, we don't have to use the MessageFilter anymore, so I bind the mojo-IPC directly in main-thread. So the Mojo IPC will
come in on the IO thread (as all Mojo IPC does) and then end up posting tasks to
the main-thread that PowerMonitorBroadcastSource is running on.

I think all the main thread does is that it posts notifications to PowerObserver instance on whatever thread the observer was added on. These power notifications are actually consumed on the watchdog thread. 

The main thread is the bottleneck and we should avoid it as much as possible.

How can the dependency on the main thread be avoided? In the GPU process, would it be legal to instantiate the PowerMonitorBroadcastSource on the watchdog thread?

Both before and after the mojofication, PowerMonitorBroadcastSource always runs on main-thread. So I guess maybe this bug is not caused by 647247?

The PowerMonitorBroadcastSource is created in ChildThreadImpl, seems in ChildThreadImpl we cannot get the gpu-watchdog-thread. And it is complex to just change its host thread in GPU process only and regardless of other processes.

Issue 746272 has been merged into this issue.

Issue 746280 has been merged into this issue.

I spent some time looking at the code trying to figure if PowerMonitorBroadcastSource could be bound to gpu watchdog thread. The idea is to make creation of PowerMonitor optional in ChildThreadImpl, and then in the case of GPU process create it later in either GpuChildThread or GpuWatchdogThread.

Creating it in GpuWatchdogThread seemed like a better idea because we want to post the actual creation on the thread runner. I coded the whole thing but ran into a layering issue because GpuWatchdogThread cannot see service_manager::Connector.

This could probably be solved either by moving GpuWatchdogThread to content or by providing some other indirection mechanism like passing a factory object to GpuWatchdogThread that would allow GpuWatchdogThread to create PowerMonitor or PowerMonitorBroadcastSource without having to know all the implementation details.

Another though that I had is that perhaps PowerMonitor should always be created on IO thread for all types of child processes. PowerMonitor is very lightweight and doesn't do much other than redirecting notifications so it makes little sense to host it in the main thread. This seems like it could be a more elegant solution that wouldn't depend on whether watchdog thread is running.

I could give this a try.

Issue 719937 has been merged into this issue.

Thanks Stanis for offering to help fix this.

If it doesn't break any other / new assumptions in Chrome's code base to host the PowerMonitor on the IO thread (and if that can be done portably) that sounds OK to me, but keep in mind I don't really know the new invariants and best practices that Mojo has introduced.

Any update on this? It's become a pressing issue for some customers, who are seeing the "Rats!" infobar far more often than they should be.

Users experienced this crash on the following builds:

Mac Dev 61.0.3159.5 -  0.28 CPM, 3 reports, 3 clients (signature [GPU hang] gl::GLImageIOSurface::BindTexImageWithInternalformat)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

[Stability Sheriff] Bug 752831 is marked Release-Block-Stable and is marked for M-62 so marking this one the same (assuming this is the root cause bug).

stanisc@: Can you please give an update on the next steps here? Is the approach in C#25 something you can implement for M-62?

Pinging as the current stability sheriff -- it's been a few weeks now, seeking an update from current owner.

I've just returned from a 2 week vacation. I started experimenting with a solution before leaving. I'll update the status later today or tomorrow.

Friendly ping to get an update on this bug as it is marked as a stable blocker.

Thanks..!

I have this coded and tested locally. Here is what I currently have:
https://chromium-review.googlesource.com/c/chromium/src/+/650887

I can't submit this yet until I fix a unit test in power_monitor_message_broadcaster_unittest.cc. The test implementation was tightly coupled to the previous implementation of PowerMonitorBroadcastSource which I changed. So it looks like I'd need to significantly re-implement the test.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/79d8fbf7cbc35d479177d17eea4fb0b822bb6ddd

commit 79d8fbf7cbc35d479177d17eea4fb0b822bb6ddd
Author: Stanislav Chiknavaryan <stanisc@chromium.org>
Date: Fri Sep 08 21:30:01 2017

Move actual implementation of PowerMonitorBroadcastSource to I/O thread

Currently PowerMonitorBroadcastSource runs on main thread in every
child process. When a power message received in the mojo layer it is
first handled on I/O thread, then gets on the main thread where it is
handled by PowerMonitorBroadcastSource which triggers a power
notification through PowerMonitor. Finally PowerMonitor broadcasts the
notification to all observers on a thread that each observer subscribed
to notifications.

There are two problems here:
1) Unnecessary thread hop from I/O thread to main thread just to 
   trigger a notification which might be eventually processed on a 
   different thread. For example, GPU Watchdog receives power 
   notification on the GPU watchdog thread.
2) The main thread tends to be a bottleneck. If there are a few
   expensive tasks queued on the main thread, the power notification
   would be queued behind them and may arrive too late. When a
   machine resumes from sleep there is a lot of I/O reading to reload
   all binary images back to memory. That is exactly the moment where
   GPU watchdog needs to be aware of the slow progress caused by the
   I/O activity and disable crashing on the timeout. But the power
   message likely gets stuck on the main thread behind the tasks that
   take long time to execute due to all the I/O so the watchdog 
   doesn't get a chance to adjust its logic.

The fix is to bypass the main thread altogether. With this change
mojo binding is done on I/O thread. PowerMonitorBroadcastSource 
handles incoming power messages and triggers PowerMonitor on I/O thread.
PowerMonitor still propagates power notification to the right threads so
there should be no difference for power observers.

I tested this change in debugger - verified that PowerMonitorBroadcastSource
code is now called on I/O thread and that a consumer such as 
HighResolutionTimerManager is still called on the main thread.

Bug: 729686
Change-Id: Iec1793682d10805674e3f2d17e27542af99fb252
Reviewed-on: https://chromium-review.googlesource.com/650887
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Reviewed-by: Brandon Jones <bajones@chromium.org>
Reviewed-by: Nasko Oskov <nasko@chromium.org>
Commit-Queue: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#500686}
[modify] https://crrev.com/79d8fbf7cbc35d479177d17eea4fb0b822bb6ddd/content/child/child_thread_impl.cc
[modify] https://crrev.com/79d8fbf7cbc35d479177d17eea4fb0b822bb6ddd/services/device/power_monitor/power_monitor_message_broadcaster_unittest.cc
[modify] https://crrev.com/79d8fbf7cbc35d479177d17eea4fb0b822bb6ddd/services/device/public/cpp/power_monitor/power_monitor_broadcast_source.cc
[modify] https://crrev.com/79d8fbf7cbc35d479177d17eea4fb0b822bb6ddd/services/device/public/cpp/power_monitor/power_monitor_broadcast_source.h
[modify] https://crrev.com/79d8fbf7cbc35d479177d17eea4fb0b822bb6ddd/services/device/public/cpp/power_monitor/power_monitor_broadcast_source_unittest.cc

My understanding is that PowerMonitorBroadcastSource should now post power notifications directly from I/O thread to GPU watchdog thread bypassing the main thread. Looking at the [GPU hang] crash rate I don't see a significant improvement but it might take a few days to take effect.

I am going to resolve this for now. I don't see much impact on the overall rate of all [GPU hang] crashes but it is difficult to isolate crashes related to suspend/resume from everything else. I looked at a number of recent [GPU hang] crashes but didn't any with a suspend/resume timestamps.

Feel free to reactivate if the problem persists. When reactivating please include the crash ID.

[Auto-generated comment by a script] We noticed that this issue is targeted for M-62; it appears the fix may have landed after branch point, meaning a merge might be required. Please confirm if a merge is required here - if so add Merge-Request-62 label, otherwise remove Merge-TBD label. Thanks.

Thanks for the fix - can you please confirm if this needs to be merged into M62? This is currently marked as a release blocker for Stable.

This bug requires manual review: M62 has already been promoted to the beta branch, so this requires manual review
Please contact the milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), bhthompson@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

 Issue 689350  has been merged into this issue.

This bug seems to have been around for a while and it doesn't seem urgent to take it in for M62. My recommendation is to wait until M63. If you disagree, please re-apply merge-request label with justification. 

Just to update:

Magic Signature: '[GPU hang] gl::GLImageIOSurface::BindTexImageWithInternalformat'

This is top#5th GPU process crash seen on latest dev-64.0.3269.3 & still seeing 11 instances from 11 clients so far.

64.0.3269.3	0.02%	11	-Dev
63.0.3239.59	0.01%	4	-Beta
62.0.3202.94	7.79%	3669	-Stable

Link to the list of builds:
--------------------------
https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Mac%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27gpu-process%27%20AND%20custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BGPU%20hang%5D%20gl%3A%3AGLImageIOSurface%3A%3ABindTexImageWithInternalformat%27&sql_dialect=dremelsql&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D#productversion:1000

As owner is not in chrome any more, could some one from dev cc'ed please take a look.

Thanks..!

kbr@, since stanisc@ is no longer on Chrome, can you suggest someone to take a look at this? Thanks!

Marking M-63 and M-64

Regarding #54, I wanted to mention that GPU hang crashes happen for a number of reasons. This bug was specific to a resuming from sleep, but it is quite possible to match the same signature in a hang unrelated to suspend/resume. In other words  crashes mentioned in #54 don't necessarily apply to this specific bug.

Cl landed at #44 is already in M63 and this bug seems to have been around for a while. We're only 5 days away for M63 stable, I'm not going to block M63 stable for this. Please let me know ASAP if there is any concern here.

Crashes observed on M63:
63.0.3239.59	0.03%	16	
63.0.3239.52	0.04%	19	
63.0.3239.40	0.09%	46	
63.0.3239.30	0.07%	33	
63.0.3239.18	0.06%	28	
63.0.3239.9	0.02%	8	
63.0.3239.5	0.00%	1	
63.0.3239.3	0.00%	2	

Removing R-B-S. This shouldn't block M63.

Desktop sheriff here -

I don't see a simple way to assess the impact of the fix in C#44.  Crash aggregates these crash dumps by magic signature and it seems like GPU hangs are distributed among several magic signatures.   Maybe there is a separate dashboard for this somewhere?  Otherwise it will require going to the data source directly.

I looked for any UMA logged for hangs in gpu_watchdog_thread.cc and didn't see any (maybe there is something captured on the browser side)?

Maybe someone on the GPU team could look at representative crash dumps and see if any have sleep/resume timing info indicating this is still an issue.

Also, can the original reporter see if it fixes their issue with sleep/resume.

For now, removing from sheriff queue as I think the right folks have looked at this and a patch has landed to resolve the original issue.

Issue 737834 has been merged into this issue.

Issue 791977 has been merged into this issue.

Unfortunately it looks like we're still seeing GPU hangs that look suspect. In Issue 791977 these two stack traces were reported. The main thread's obviously idle, so there's no excuse for the watchdog firing.

I don't have any ideas about how to resolve this given the fix from https://chromium-review.googlesource.com/650887 . Does anyone have ideas about why this might still be happening during suspend/resume?

Thread 0 (id: 3541) MAGIC SIGNATURE THREAD
Stack Quality81%Show frame trust levels
0x00007fff59009e76	(libsystem_kernel.dylib + 0x00012e76 )	mach_msg_trap
0x00007fff318e25d4	(CoreFoundation + 0x000865d4 )	__CFRunLoopServiceMachPort
0x00007fff318e1926	(CoreFoundation + 0x00085926 )	__CFRunLoopRun
0x00007fff318e0fa2	(CoreFoundation + 0x00084fa2 )	CFRunLoopRunSpecific
0x00007fff339913f5	(Foundation + 0x000213f5 )	-[NSRunLoop(NSRunLoop) runMode:beforeDate:]
0x000000010e9e9a7d	(Google Chrome Framework -message_pump_mac.mm:724 )	base::MessagePumpNSRunLoop::DoRun(base::MessagePump::Delegate*)
0x000000010e9e885d	(Google Chrome Framework -message_pump_mac.mm:179 )	base::MessagePumpCFRunLoopBase::Run(base::MessagePump::Delegate*)
0x000000010ea0b9e3	(Google Chrome Framework -run_loop.cc:114 )	<name omitted>
0x000000011277dda2	(Google Chrome Framework -gpu_main.cc:335 )	content::GpuMain(content::MainFunctionParams const&)
0x000000010e5e337e	(Google Chrome Framework -content_main_runner.cc:705 )	content::ContentMainRunnerImpl::Run()
0x000000010feb396a	(Google Chrome Framework -main.cc:456 )	service_manager::Main(service_manager::MainParams const&)
0x000000010e5e2933	(Google Chrome Framework -content_main.cc:19 )	content::ContentMain(content::ContentMainParams const&)
0x000000010cc8996e	(Google Chrome Framework -chrome_main.cc:125 )	ChromeMain
0x000000010cc6449b	(Google Chrome Helper -chrome_exe_main_mac.cc:165 )	main
0x00007fff58ec3144	(libdyld.dylib + 0x00001144 )	start
Thread 1 (id: 3854) CRASHED [EXC_BAD_ACCESS / KERN_INVALID_ADDRESS @ 0x00000000 ]
Stack Quality77%Show frame trust levels
0x000000010fb483ea	(Google Chrome Framework -gpu_watchdog_thread.cc:421 )	gpu::GpuWatchdogThread::DeliberatelyTerminateToRecoverFromHang()
0x000000010e9c1ecb	(Google Chrome Framework -callback.h:65 )	base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*)
0x000000010e9e70e3	(Google Chrome Framework -message_loop.cc:394 )	base::MessageLoop::RunTask(base::PendingTask*)
0x000000010e9e7753	(Google Chrome Framework -message_loop.cc:406 )	base::MessageLoop::DoDelayedWork(base::TimeTicks*)
0x000000010e9e9432	(Google Chrome Framework -message_pump_mac.mm:456 )	base::MessagePumpCFRunLoopBase::RunWork()
0x000000010e9da369	(Google Chrome Framework + 0x01d54369 )	base::mac::CallWithEHFrame(void () block_pointer)
0x000000010e9e8d3e	(Google Chrome Framework -message_pump_mac.mm:428 )	base::MessagePumpCFRunLoopBase::RunWorkSource(void*)
0x00007fff318ff820	(CoreFoundation + 0x000a3820 )	__CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__
0x00007fff319b94cb	(CoreFoundation + 0x0015d4cb )	__CFRunLoopDoSource0
0x00007fff318e22bf	(CoreFoundation + 0x000862bf )	__CFRunLoopDoSources0
0x00007fff318e173c	(CoreFoundation + 0x0008573c )	__CFRunLoopRun
0x00007fff318e0fa2	(CoreFoundation + 0x00084fa2 )	CFRunLoopRunSpecific
0x000000010e9e97ce	(Google Chrome Framework -message_pump_mac.mm:670 )	base::MessagePumpCFRunLoop::DoRun(base::MessagePump::Delegate*)
0x000000010e9e885d	(Google Chrome Framework -message_pump_mac.mm:179 )	base::MessagePumpCFRunLoopBase::Run(base::MessagePump::Delegate*)
0x000000010ea0b9e3	(Google Chrome Framework -run_loop.cc:114 )	<name omitted>
0x000000010ea3d76a	(Google Chrome Framework -thread.cc:338 )	base::Thread::ThreadMain()
0x000000010ea38086	(Google Chrome Framework -platform_thread_posix.cc:75 )	base::(anonymous namespace)::ThreadFunc(void*)
0x00007fff5914d6c0	(libsystem_pthread.dylib + 0x000036c0 )	_pthread_body
0x00007fff5914d56c	(libsystem_pthread.dylib + 0x0000356c )	_pthread_start
0x00007fff5914cc5c	(libsystem_pthread.dylib + 0x00002c5c )	thread_start
0x000000010ea3802f	(Google Chrome Framework + 0x01db202f )

Users experienced this crash on the following builds:

Mac Beta 65.0.3325.106 -  0.60 CPM, 31 reports, 20 clients (signature [GPU hang] gl::GLImageIOSurface::BindTexImageWithInternalformat)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Users experienced this crash on the following builds:

Mac Dev 66.0.3359.26 -  0.51 CPM, 3 reports, 3 clients (signature [GPU hang] base::MessagePumpNSRunLoop::DoRun)

If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Just to update the latest behavior of this issue in the latest channels for the magic signature - [GPU hang] base::MessagePumpNSRunLoop::DoRun, which is merged into this issue in C#61

Still seeing 95 crashes from 94 clients so far on latest Stable - 69.0.3497.81 on Mac OS. This crash is ranked as #20 in 'GPU-Process' Stable crashes. 

71.0.3546.0	0.00%	1 - Previous Canary
70.0.3538.9	0.01%	4 - Dev
69.0.3497.81	0.25%	101- Stable/Beta

Link to the list of builds:
-------------------------
https://crash.corp.google.com/browse?q=product_name%3D%27Chrome_Mac%27+AND+expanded_custom_data.ChromeCrashProto.ptype%3D%27gpu-process%27+AND+expanded_custom_data.ChromeCrashProto.magic_signature_1.name%3D%27%5BGPU+hang%5D+base%3A%3AMessagePumpNSRunLoop%3A%3ADoRun%27#-productname:1000,productversion:1000,-magicsignature:50,-magicsignature2:50,-stablesignature:50,-magicsignaturesorted:50

Thanks!

Hello!
This bug is receiving this notice because there has been no acknowledgment of its existence in quite a bit of time
- If you are currently working on this bug, please provide an update.
- If you are currently affected by this bug, please update with your current symptoms and relevant logs.

If there has been no updates provided by EOD Wednesday, 12/19/18 (5pm EST), this bug will be archived and can be re-opened at any time deemed necessary.

Thank you!

Please don't archive this bug. We intend to do tests of the GPU watchdog on macOS to make sure it's behaving properly.

Thank you for the update. Removing from list.