We received this as an external report from our customer forums, and a number of users describe games and video content hanging as they switch tabs.

I can consistently reproduce this under stressful conditions in a VMWare Fusion VM running Windows 10, but we don't generally see it on fast machines.  It looks like the issue has a timing component, and is more apparent on slow hardware.

Also, there's a workaround for this, in that resizing the HTML causes Chrome to resume the rendering workflow.

The following JavaScript works to address this, but it's not ideal:

document.addEventListener('visibilitychange', function(){ 

            if(!document.hidden) {
                document.querySelector('html').style.width="99.9%";

                setTimeout( function(){
                    document.querySelector('html').style.width="100%";
                }, 1000);
            }
        });

Also able to replicate this issue inside the ChromeOS on a HP Chromebook 14 G5 with the Celeron N3350 processor with 4GB RAM and a 16GB SSD.  Resizing the browser does seem to resolve it; however, without access to the web source we must manually resize the browser which is only an inconvenient work around.

This seems quite bad.  Bumping to P2, but maybe should be even P1.

I couldn't repro on linux.

I couldn't reproduce on my desktop Win10, nor on my laptop Win10. Maybe both of them are not slow enough?

I apologize, my previous comment was not correct.  I am UNABLE to reproduce the issue with the video listed above on the Chromebook; however, I AM able to reproduce the bug inside any Adobe Connect meeting room.  The room will load fine, I can see chat messages flowing, videos playing, web cams working; however, when switching tabs (using CTRL+TAB or just clicking another tab) and returning, the entire room is hung up until we resize the window.

This issue (which I am assuming is the same bug listed here) occurs with a 100% fail rate on over 100 of the following device.

ChromeOS: 69.0.3497.120 (Official Build)(64-bit)
Computer: HP Chromebook 14 G5
Processor: Celeron N3350
Memory: 4GB RAM

Sounds like it could be a frame eviction issue. +samans@ in case he has ideas.

I'm not sure if this is standard protocol to write as a comment; however, I was hoping to check in to see if there is a time frame for this fix?  The original bug was reported almost 2 months ago, and it appears that there hasn't been much conversation in the past month.

Being an public K12 school, we utilize Chromebooks widely and prefer to use Chrome as our school's browser.  The issue is directly affecting the education of our children, so it is important I stay up on this issue.

Adding a few ChromeOS folks

It's hard for us to investigate and fix an issue if we can't reproduce.

Could someone who can reproduce this issue record a trace?

1) Close unnecessary tabs, goto about:tracing, click record, select "rendering", and start recording/
2) reproduce the issue on new tabs
3) go back to the tracing tab, stop recording, and save the data, upload it to this bug.

Here is the recorded trace and the steps we took to produce it (in the event we messed up).

1) Opened about:tracing
2) Opened an Adobe Connect room in a new tab, had the room running
3) Went back to about:tracing, clicked record, rendering, record
4) Switched back to the Adobe Connect room (it was already frozen because we switched tabs)
5) Resized the window to allow the class to render correctly again
6) Switched to a new tab
7) Switched back to the Adobe Connect tab (it was frozen again)
8) Resized the browser to allow it to render correctly
9) Switched back to about:tracing
10) Ended recording.

This was done on a Chromebook:

ChromeOS: 70.0.3538.76 (Official Build)(64-bit)
Computer: HP Chromebook 14 G5
Processor: Celeron N3350
Memory: 4GB RAM

Deleted: trace_flashbug.json.gz 13.1 MB

trace_flashbug.json.gz 13.1 MB Download

Thanks. I'll take a look at the trace.

It looks like pepper main thread tasks take a really long time (500ms+).  Unfortunately rendering categories don't emit enough information about this.  Can you record another trace, but this time click "Edit Categories" and click "All" under "Record Categories" (not "Disabled by Default Categories")?

Here is the new run with "All" selected under "Record Categories".  Please let me know if I can assist in any other way.

Deleted: trace_NewTrace.json.gz 12.4 MB

trace_NewTrace.json.gz 12.4 MB Download

Looks like those tasks are just callbacks into the pepper plugin (flash).  piman@ can you take a look at the trace to check if anything stands out?

"CallOnMainThread" is the pepper equivalent to PostTask, so the 500+ms jank is Flash posting to itself. It does end up calling command buffer flush in the instances that I saw. I don't know whether or not that is related to the issue.

We had another Chromebook on hand as well.  It is the same model number; however, has twice the amount of memory.  We still see a 100% fail on a larger room with 15+ people and multiple pods open.

So I'm assuming that the amount of data being pushed by flash is directly affecting the fail rate of the computer.  While the simple flash video in the original bug report may work fine on one computer model, something more complex like an Adobe Connect room may not.  Regardless of the source flash file you are attempting to view, the problem and fail does seem to be the same across all computer models:

Flash loads up fine, switch a tab, come back, the audio is still live and working well; however, the entire flash screen is froze up.  This is true even in a live environment such as an Adobe Connect meeting room.  Resizing the web browser resolves the freeze.

This test was performed with the following:
ChromeOS: 70.0.3538.76 (Official Build)(64-bit)
Computer: HP Chromebook 14 G5
Processor: Celeron N3350
Memory: 8GB RAM

We are continuing to troubleshoot as much as we can here.  I am interested to see if the amount of memory makes any difference on the freeze threshold.  

OK, I took a careful look at the traces and saw everything fit into the 10 steps you described. I also saw the flush stopped at tab switch and didn't resume when switched back. However, I still didn't get the why.

I think it's beneficial for jeclark@adobe.com to record two traces with VM, one with 2-core where this freeze happens, and one with 4-core where things work fine, so I can tell what's missing by comparing. I already send him the detailed instructions through email.

The mentioned video is so far the easiest way to reproduce the problem.

When Flash Player submits a framebuffer, it calls the following API:

pp::Graphics3D::SwapBuffers(const CompletionCallback& cc)

In normal case, Flash Player receives the completion callback indicating it is ready to submit the next framebuffer. When the problem happens, the completion callback never comes back, so the video freezes while audio continues.

I think it would much more efficient to set up a basic VM with Win10 RS5 64-bit, so that the above observation could be evaluated directly.

Ok, so I think this is a bug in cc in the proxy code, where the early-out if !visible loses state.

On SwapBuffers, the pepper code sets a new texture on the TextureLayer (via TextureLayer::SetTransferableResource) which requires a commit. The Ack comes (indirectly) from RenderWidget::DidCommitAndDrawCompositorFrame, i.e. after the commit has happened and we've drawn a frame.

What happens when things get blocked:
1- TextureLayer::SetTransferableResource -> LayerTreeHost::SetNeedsCommit -> ProxyMain::SetNeedsCommit, sets max_requested_pipeline_stage_ to COMMIT_PIPELINE_STAGE and posts a task (A) to the impl thread to request a BeginMainFrame.
2- RenderWidget becomes invisible, calls LayerTreeHost::SetVisible(false), posts a task (B) to the impl thread to set invisible.
3- on the impl thread before B runs but after A, we get a BeginFrame, we then post to the main thread to do it.
4- ProxyMain::BeginMainFrame runs, final_pipeline_stage_ gets updated to COMMIT_PIPELINE_STAGE, max_requested_pipeline_stage_ gets updated to NO_PIPELINE_STAGE, but because the LayerTreeHost is not visible, we take the early out (EarlyOut_NotVisible). We abort the commit, so we don't run DidCommitAndDrawCompositorFrame (but we note that we need a BeginMainFrame again).

[time passes]

5- RenderWidget becomes visible again, calls LayerTreeHost::SetVisible(true), posts a task to the impl thread to set visible. This resumes BeginFrames
6- BeginFrame happens, we post to the main thread
7- ProxyMain::BeginMainFrame runs. max_requested_pipeline_stage_ is still NO_PIPELINE_STAGE, so final_pipeline_stage_ is set to NO_PIPELINE_STAGE. If we have no other layer updates then we early-out and abort the commit (EarlyOut_NoUpdates). We abort the commit, so we don't run DidCommitAndDrawCompositorFrame, we don't pass Go and don't collect $200.

The fundamental problem is that max_requested_pipeline_stage_ gets updated to NO_PIPELINE_STAGE in step 4. For deferred commits early outs we keep track of the final_pipeline_stage_ in deferred_final_pipeline_stage_ for subsequent runs, but we don't do anything similar for !visible early outs.
I suspect we could move the !layer_tree_host_->IsVisible() early out to much earlier in the function (before we touch final_pipeline_stage_/max_requested_pipeline_stage_) though I'm not an expert in the area.

I agree, and suspect unit tests would tell us if it's wrong.

+boliu@ FYI since you saw something similar in webview

Thanks piman. I'll upload a potential fix and see how the bots react to that.

jeclark@adobe.com, xzhang@adobe.com: I have a potential fix (thanks for piman@'s analysis). However, I have no luck in reproducing the issue in a Windows VM after trying various cores/execution caps.

So can you help verifying the patch? If yes, I will share two links. One is chromium without the fix, that you should be able to reproduce the issue on your VM. The other is chromium with the fix, that the issue should no longer exist.

zmo@ Sure, we can help verifying your fix.

Thank you. I shared the two builds with you two through Google drive. Please let me know after you test them.

zmo@ I have tested the 2 Chromium builds on my VM. I was able to see the difference between them:

without the fix: I can easily reproduce the issue, that is, video freezes, while audio continues, and it cannot recover by itself.

with the fix: both video and audio will stop after the tab switching for a little while; If I stop tab switching, the video and audio will resume shortly.

From my point of view, I think this fix has big improvement, I would accept that it fixes the frozen video rendering issue triggered by fast tab switching.

I hope it can solve the issues post by other users for different use cases.

Thanks for confirming. I'll land the fix soon.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/5b30cf37b8479fc60c52a5b98e60d4bfa9384ebc

commit 5b30cf37b8479fc60c52a5b98e60d4bfa9384ebc
Author: Zhenyao Mo <zmo@chromium.org>
Date: Sat Dec 01 00:31:25 2018

In ProxyMain::BeginMainFrame, check visibility before setting stage states.

Otherwise if there is a commit request and we early out due to being invisible
before commiting, and we set the max+requested_pipeline_stage_ to
NO_PIPELINE_STAGE, then that commit is lost. For flash this results in a commit
never receiving callback, and the next commit can never be requested, thus
rendering is stalled.

BUG= 890008 
TEST=cc_unittests
R=piman@chromium,enne@chromium.org,danakj@chromium.org

Change-Id: I240be7bb5496f28c0217ee7ef86533509bafcd16
Reviewed-on: https://chromium-review.googlesource.com/c/1343048
Commit-Queue: Zhenyao Mo <zmo@chromium.org>
Reviewed-by: enne <enne@chromium.org>
Reviewed-by: Sunny Sachanandani <sunnyps@chromium.org>
Cr-Commit-Position: refs/heads/master@{#612872}
[modify] https://crrev.com/5b30cf37b8479fc60c52a5b98e60d4bfa9384ebc/cc/test/layer_tree_test.cc
[modify] https://crrev.com/5b30cf37b8479fc60c52a5b98e60d4bfa9384ebc/cc/test/test_hooks.h
[modify] https://crrev.com/5b30cf37b8479fc60c52a5b98e60d4bfa9384ebc/cc/trees/layer_tree_host_impl.h
[modify] https://crrev.com/5b30cf37b8479fc60c52a5b98e60d4bfa9384ebc/cc/trees/layer_tree_host_unittest_proxy.cc
[modify] https://crrev.com/5b30cf37b8479fc60c52a5b98e60d4bfa9384ebc/cc/trees/proxy_impl.cc
[modify] https://crrev.com/5b30cf37b8479fc60c52a5b98e60d4bfa9384ebc/cc/trees/proxy_main.cc
[modify] https://crrev.com/5b30cf37b8479fc60c52a5b98e60d4bfa9384ebc/cc/trees/single_thread_proxy.cc

Your change meets the bar and is auto-approved for M72. Please go ahead and merge the CL to branch 3626 manually. Please contact milestone owner if you have questions.
Owners: govind@(Android), kariahda@(iOS), djmm@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/d4ae5c3e505a65dfbacd35c42b2d753bf3320d49

commit d4ae5c3e505a65dfbacd35c42b2d753bf3320d49
Author: Zhenyao Mo <zmo@chromium.org>
Date: Mon Dec 03 21:13:24 2018

In ProxyMain::BeginMainFrame, check visibility before setting stage states.

Otherwise if there is a commit request and we early out due to being invisible
before commiting, and we set the max+requested_pipeline_stage_ to
NO_PIPELINE_STAGE, then that commit is lost. For flash this results in a commit
never receiving callback, and the next commit can never be requested, thus
rendering is stalled.

BUG= 890008 
TEST=cc_unittests
R=​piman@chromium,enne@chromium.org,danakj@chromium.org

Change-Id: I240be7bb5496f28c0217ee7ef86533509bafcd16
Reviewed-on: https://chromium-review.googlesource.com/c/1343048
Commit-Queue: Zhenyao Mo <zmo@chromium.org>
Reviewed-by: enne <enne@chromium.org>
Reviewed-by: Sunny Sachanandani <sunnyps@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#612872}(cherry picked from commit 5b30cf37b8479fc60c52a5b98e60d4bfa9384ebc)
Reviewed-on: https://chromium-review.googlesource.com/c/1359312
Reviewed-by: Zhenyao Mo <zmo@chromium.org>
Cr-Commit-Position: refs/branch-heads/3626@{#14}
Cr-Branched-From: d897fb137fbaaa9355c0c93124cc048824eb1e65-refs/heads/master@{#612437}
[modify] https://crrev.com/d4ae5c3e505a65dfbacd35c42b2d753bf3320d49/cc/test/layer_tree_test.cc
[modify] https://crrev.com/d4ae5c3e505a65dfbacd35c42b2d753bf3320d49/cc/test/test_hooks.h
[modify] https://crrev.com/d4ae5c3e505a65dfbacd35c42b2d753bf3320d49/cc/trees/layer_tree_host_impl.h
[modify] https://crrev.com/d4ae5c3e505a65dfbacd35c42b2d753bf3320d49/cc/trees/layer_tree_host_unittest_proxy.cc
[modify] https://crrev.com/d4ae5c3e505a65dfbacd35c42b2d753bf3320d49/cc/trees/proxy_impl.cc
[modify] https://crrev.com/d4ae5c3e505a65dfbacd35c42b2d753bf3320d49/cc/trees/proxy_main.cc
[modify] https://crrev.com/d4ae5c3e505a65dfbacd35c42b2d753bf3320d49/cc/trees/single_thread_proxy.cc

The following revision refers to this bug: 
https://chromium.googlesource.com/chromium/src.git/+/d4ae5c3e505a65dfbacd35c42b2d753bf3320d49

Commit: d4ae5c3e505a65dfbacd35c42b2d753bf3320d49
Author: zmo@chromium.org
Commiter: zmo@chromium.org
Date: 2018-12-03 21:13:24 +0000 UTC

In ProxyMain::BeginMainFrame, check visibility before setting stage states.

Otherwise if there is a commit request and we early out due to being invisible
before commiting, and we set the max+requested_pipeline_stage_ to
NO_PIPELINE_STAGE, then that commit is lost. For flash this results in a commit
never receiving callback, and the next commit can never be requested, thus
rendering is stalled.

BUG= 890008 
TEST=cc_unittests
R=​piman@chromium,enne@chromium.org,danakj@chromium.org

Change-Id: I240be7bb5496f28c0217ee7ef86533509bafcd16
Reviewed-on: https://chromium-review.googlesource.com/c/1343048
Commit-Queue: Zhenyao Mo <zmo@chromium.org>
Reviewed-by: enne <enne@chromium.org>
Reviewed-by: Sunny Sachanandani <sunnyps@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#612872}(cherry picked from commit 5b30cf37b8479fc60c52a5b98e60d4bfa9384ebc)
Reviewed-on: https://chromium-review.googlesource.com/c/1359312
Reviewed-by: Zhenyao Mo <zmo@chromium.org>
Cr-Commit-Position: refs/branch-heads/3626@{#14}
Cr-Branched-From: d897fb137fbaaa9355c0c93124cc048824eb1e65-refs/heads/master@{#612437}