Can we get more clarity on this issue?

What OS's is this occuring on? Original reports all seem to be on ChromeOS. But #5seems to indicate other OS's as well. But all reports seem to be for Chrome OS. Can someone please confirm?

marcore@ / kathrelkeld@ - are you able to replicate this in test lab?

yanglee@ - can we please test this in enterprise lab as well?

Citrix Comment- this is a Chrome Os issue and not a chrome browser issue. tried Chrome browser version - 67.0.3396.99 (Official Build) (64-bit) and 68.0.3440.84 (Official Build) (64-bit) on Mac OS.
Reproduced on Chrome OS version 67 and 68.

I believe that comment #11 was from kathrelkeld@'s testing lab (although please correct me if I'm wrong), so I believe this issue can be reproduced on Chrome OS. Judging from comment #16, it is likely an OS issue and not a browser issue, as it is not reproducible on Mac OS.

Thanks for more info!

sona.somaraokulkarni@ - can you please try recreating this on Windows as well? Since Windows is marked, I'd like to confirm. 

Checked on Windows with Chrome version Version 68.0.3440.84 (Official Build) (64-bit) and issue isn't happening.\
But is reproduced only on Chrome OS 67 , 68
Hence looks like it is a Chrome OS issue

Citrix comment- we could reproduce the issue even with a sample app.(using setShape API).

Please find below the steps to reproduce the same:
1. Install the sample app (attached)
2. Open any three apps in windowed mode (eg: Files, caret, text)
3. Launch sample (setshape app) it will create 2 windows 
4. click setshape button 
5. Click on the green window
Observe the windows going blank in a cascading manner (as we click app windows)

See attached video for clarity.: https://citrix.sharefile.com/d-sfec1fbd624c4bd1a

Deleted: setshape_sample_app.zip 1.8 KB

setshape_sample_app.zip 1.8 KB Download

OK, seeing the following in the logs:

[5390:5390:0806/162741.633614:ERROR:gles2_cmd_decoder.cc(18055)] [.DisplayCompositor-0x20dc4d3a4a00]GL ERROR :GL_INVALID_OPERATION : glCreateAndConsumeTextureCHROMIUM: invalid mailbox name
[5390:5390:0806/162741.633971:ERROR:gles2_cmd_decoder.cc(10173)] [.DisplayCompositor-0x20dc4d3a4a00]RENDER WARNING: texture bound to texture unit 0 is not renderable. It maybe non-power-of-2 and have incompatible texture filtering.
[5390:5390:0806/162741.730889:ERROR:gles2_cmd_decoder.cc(10173)] [.DisplayCompositor-0x20dc4d3a4a00]RENDER WARNING: texture bound to texture unit 0 is not renderable. It maybe non-power-of-2 and have incompatible texture filtering.

This *seems* like a GPU issue - not sure who owns the GPU stack on cros, but CCing kbr and vmiura

If anyone at Google needs a test environment to recreate, feel free to ping me as I have a full Citrix stack running.

I'm able to reproduce with the Citrix App (setshape) on 67.0.3396.87 and 67.0.3396.99 nyan_blaze
with 66.0.3359.181 there is no white app.
M66 version: https://drive.google.com/open?id=1MXYF-1_XRPPrENFgjj9CEkeR3_hAvkD7
M66 debug logs: https://drive.google.com/open?id=1K1K2mGE3KnqbxgmjAGWVs2r3hHTOonkV
after upgrade to 67.0.3396.99 (around 15:25)
M67 version: https://drive.google.com/open?id=1cD1slFFzedz2XpJclBWlvyIk6CnpmIAw
M67 debug logs: https://drive.google.com/open?id=1X6j2Dt5GbhDZLJSoPCkeOvbOk56rZ36K
in the M67 log I see:
[988:1220:0807/152556.795894:ERROR:service_manager_context.cc(258)] Attempting to run unsupported native service: /opt/google/chrome/chrome_renderer.service

I was able to bisect:
last known good:
67.0.3383.0/10533.0.0 2018-03-31 03:16
first bad:
67.0.3383.0/10534.0.0 2018-03-31 11:12
https://crosland.corp.google.com/log/10533.0.0..10534.0.0
Please help find the right commit.

+ Geetha & Song

The warnings about the DisplayCompositor in #22 above are probably not related, unless they aren't seen on the good version of the browser on ChromeOS. +enne

Looking at the CrOS changelog it doesn't look like any of those commits could be the cause. Can we get a browser-side regression range as well?

Able to reproduce "comment#20" on 67.0.3383.0/10533.0.0 - Peppy

Citrix Comment- When can this be fixed? We need to release our product and most of our customers are complaining about this issue.

sorry for the previous incorrect bisect(it was my first on ChromeOS), I've remade it on a different HW and tested it more:
Last good: 67.0.3381.0/10526.0.0 2018-03-28 19:19
first bad: 67.0.3383.0/10527.0.0 2018-03-29 03:14

https://crosland.corp.google.com/log/10526.0.0..10527.0.0

It's not HW depended, I've tested on nyan-blaze and veyron-minnie.
with the test app Citrix uploaded in this bug, I can only test on ChromeOs.

kathrelkeld@ is probably not the right owner for this bug.

Also, given that the setShape API is suspected, which boilds down to the views::Widget code (SetShape()), then perhaps //src/ui/views/widget/ OWNERS could help triaging this?..
sky@ - could you please take a look? Thanks.

If I'm reading comment 35 right, this corresponds to chrome revisions 67.0.3381.0 and 67.0.3383.0. Assuming I have that right, log is here: https://chromium.googlesource.com/chromium/src/+log/67.0.3381.0..67.0.3383.0?pretty=fuller&n=10000 .

François landed (and enabled) draw occlusion in that timeframe, which I suspect is the culprit here. In particular if we're incorrectly calculating occlusion, we may not think the browser is visible when in fact it is.

In particular, it might be: https://chromium.googlesource.com/chromium/src/+/e616115a9b1dca27ffdf447c79b4413305c49bb4 and https://chromium.googlesource.com/chromium/src/+/e05ba4c58bde7423528fbb71f61b74543bae919c .

If draw occlusion is the culprit, would this explain the cascading behavior shown in the video in comment #20 (where focus on each window in turn causes another window to go blank)? 

Not sure. There is a flag for draw-occlusion. You could try turning it off to see if that fixs things.

I disabled the #enable-draw-occlusion flag and was able to reproduce the blanking-out of apps after the device was restarted. Could this CL have been the culprit even if the feature is disabled? If so, perhaps some kind of interaction between the setShape API and draw occlusion?

Thanks for your help on this high priority bug.

If disabling draw occlusion doesn't help, then that isn't it.

Which Citrix Receiver is this? The android one?

No, this is the Chrome App. 

https://chrome.google.com/webstore/detail/citrix-receiver/haiffjcadagjlijoggckpgfnoeiflnem

Comment #20 also includes a sample Chrome app using the setShape API that exhibits the same problem. Sona, can you confirm that you are not seeing this issue on the Android Receiver?

Update: I tried the upcoming version of Citrix Receiver for Android (renamed Citrix Workspace) and did not observe this bug. So it is limited to the Chrome app. 

sky@, is there anything else we can do to help debug?

As disabling occlusion didn't fix it then fdoray isn't a good owner for this.

I believe this code is using ChromeNativeAppWindowViews::SetShape(), which calls to Widget::SetShape(), NativeWidgetAura::SetShape(), and then ui::Layer::SetAlphaShape(). AFAICT none of that code has changed recently, and certainly not in the timeframe of this regression.

I believe there has been work on minimizing drawing by detecting when layers are obscured by other layers. I suspect this code path is tickling a bug in that code, or something higher in the chain isn't configuring the shape correctly. Either one could lead to this.

Will talk with some of the viz folks to see if they have suggestions.

Do we know if the shape being supplied is valid? Might the issue be they are setting a bad shape (way too big) and only drawing into a portion of the window?
Here's some bad ascii art:

...................
...................
...................
...................
xxxx...............
xxxx...............
xxxx...............

The x's represent the *actual* bits being drawn into. If the shape encompasses the whole screen, then I could certainly see something like being described here happening. Maybe the coordinates are wrong?

Comment from Citrix: we tried creating a sample using setShape API and is attached in comment 20, please try using it as that shows coordinates passed.
Thanks!

The flag you used --Enable draw occlusion is the one i created my occlusion curling algorithms, cl: https://chromium.googlesource.com/chromium/src/+/e05ba4c58bde7423528fbb71f61b74543bae919c

and it's unrelated to enable/disable Francois's cl: https://chromium.googlesource.com/chromium/src/+/e616115a9b1dca27ffdf447c79b4413305c49bb4

Could you test again if it's related to Francois's change? 

my cl was originally landed in this cl: https://chromium-review.googlesource.com/c/chromium/src/+/711391 (disable by default until Mar)

it seems that the feature Francois added can be disabled by including the switch switches::kDisableBackgroundingOccludedWindowsForTesting -- disable-backgrounding-occluded-windows, at run time, Could someone check if it is fix by appending that flag?

yiyix@ with that flags is fixed:
tested on:
Chrome://version 

Google Chrome: 67.0.3383.0 (Official Build) dev (32-bit)
Revision: 0
Platform: 10527.0.0 (Official Build) dev-channel nyan_blaze
Firmware Version: Google_Nyan_Blaze.5771.63.0
User Agent: Mozilla/5.0 (X11; CrOS armv7l 10527.0.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3383.0 Safari/537.36

Command Line: /opt/google/chrome/chrome 
[CUT]
--disable-backgrounding-occluded-windows=TRUE

Deleted: M67_comment_50_About Version.pdf 75.5 KB

M67_comment_50_About Version.pdf 75.5 KB Download

Since Francois's cl caused this bug, i will pass it to him. 

Fdoray, please take immediate action on this issue.

- If the issue is being caused by flagged code can't we fix for everyone including released 67 and 68 stable builds by disabling the experiment server side? If so please do this immediately.

- Assuming above, actual fixes to the flagged occluded windows code should land in canary and (imho) not be merged down, the flag should be left disabled for stable until fixed code has gone all the way through release process and we haven't seen recurrence of these issues.

Citrix Comment:
Issue 864508: "Citrix Receiver" causing background apps to go blank – this is the first part of the issue, where the receiver screen becomes blank. Google is still trying to figure out what could be the issue.
 Issue 857427 : can't get a webview from Activity anymore – This is the bigger problem, as soon as the receiver screen becomes blank it becomes inactive as we use Webview to show published apps and dekstops, if this issue exists then one customer can launch only one app at a time. Luckily this issue was only in beta channel of Chrome 68 but was fixed in the stable channel.
We request that the bugfix for this issue with webview be available in all the forward going builds of Chrome OS.

If this was in 67, we cannot block stable on it as per comment 9.

Finch config to disable the feature is being reviewed. This will fix the issue quickly for affected user.

A code fix (that will take longer to reach users) will be uploaded for review today.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a3a0b7f4c69c833cbc4f4c6f6adcf1b21382abf2

commit a3a0b7f4c69c833cbc4f4c6f6adcf1b21382abf2
Author: Francois Doray <fdoray@chromium.org>
Date: Wed Aug 15 15:44:32 2018

aura: Add aura::WindowObserver::OnWindowAlphaShapeChanged().

aura::WindowObserver is invoked when an alpha shape is set for the
layer of a window.

In a future CL, aura::WindowOcclusionTracker will recompute occlusion
states when this method is invoked for a window in a tracked window
tree. Indeed, setting an alpha shape can affect occlusion.

Bug: 864508
Change-Id: If90d8cf143961655bc854d4217c867cc6d29e3be
Reviewed-on: https://chromium-review.googlesource.com/1173166
Reviewed-by: Sadrul Chowdhury <sadrul@chromium.org>
Commit-Queue: François Doray <fdoray@chromium.org>
Cr-Commit-Position: refs/heads/master@{#583259}
[modify] https://crrev.com/a3a0b7f4c69c833cbc4f4c6f6adcf1b21382abf2/ui/aura/window.cc
[modify] https://crrev.com/a3a0b7f4c69c833cbc4f4c6f6adcf1b21382abf2/ui/aura/window.h
[modify] https://crrev.com/a3a0b7f4c69c833cbc4f4c6f6adcf1b21382abf2/ui/aura/window_observer.h
[modify] https://crrev.com/a3a0b7f4c69c833cbc4f4c6f6adcf1b21382abf2/ui/aura/window_unittest.cc
[modify] https://crrev.com/a3a0b7f4c69c833cbc4f4c6f6adcf1b21382abf2/ui/compositor/layer.cc
[modify] https://crrev.com/a3a0b7f4c69c833cbc4f4c6f6adcf1b21382abf2/ui/compositor/layer_delegate.cc
[modify] https://crrev.com/a3a0b7f4c69c833cbc4f4c6f6adcf1b21382abf2/ui/compositor/layer_delegate.h
[modify] https://crrev.com/a3a0b7f4c69c833cbc4f4c6f6adcf1b21382abf2/ui/compositor/layer_unittest.cc

We just shipped a Finch config to disable occlusion for all ChromeOS users. This should fix this issue.

I've tested on 67.0.3396.99 and it works. (after at least one reboot)

Fdoray how can I check that the server side configuration is active on my devices ?

Deleted: M67_works_About Version.pdf 76.0 KB

M67_works_About Version.pdf 76.0 KB Download

marcore@: I copied the "Variations" section of your attached "About Version" in go/finch-hashes and confirmed that you are in the "DisableWebContentsOcclusion-OcclusionDisabled" group, which corresponds to the fix. The hashes of interest are af59fc20-2599138c.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3da2da6a07d1df3f9cd3aed22dd3dbdd2d2ca262

commit 3da2da6a07d1df3f9cd3aed22dd3dbdd2d2ca262
Author: Francois Doray <fdoray@chromium.org>
Date: Fri Aug 24 21:54:46 2018

aura: Do not allow occlusion by shaped windows.

Preivously, the shape of a window was not taken into account when
computing occlusion state. A window with a shape that didn't fully
cover its bounds could wrongly be considered to occlude another
window underneath it.

With this CL, shaped windows can't occlude other windows (similar
to windows that are animated or not axis-aligned).

Bug: 864508
Change-Id: I01ff1afda438982964cdc2d786f37caaa03677be
Reviewed-on: https://chromium-review.googlesource.com/1173318
Reviewed-by: Sadrul Chowdhury <sadrul@chromium.org>
Commit-Queue: François Doray <fdoray@chromium.org>
Cr-Commit-Position: refs/heads/master@{#586011}
[modify] https://crrev.com/3da2da6a07d1df3f9cd3aed22dd3dbdd2d2ca262/ui/aura/window_occlusion_tracker.cc
[modify] https://crrev.com/3da2da6a07d1df3f9cd3aed22dd3dbdd2d2ca262/ui/aura/window_occlusion_tracker.h
[modify] https://crrev.com/3da2da6a07d1df3f9cd3aed22dd3dbdd2d2ca262/ui/aura/window_occlusion_tracker_unittest.cc

Re-confirmed that https://chromium-review.googlesource.com/1173318 fixes the issue.

Remaining work before closing this bug: Stop the field trial that disables the WebContentsOcclusion feature.

Occlusion has been re-enabled on ChromeOS via field trial.

I verified that it didn't break the setshape app in comment #20 works properly with occlusion re-enabled.

Let me know if you observe breakages on latest canary/dev/stable releases.

Citrix Comment:

yes re-enabling occlusion has again started breaking our customers on Chrome Version 71.0.3578.49 (Official Build) beta (64-bit)
They hare facing same issue as before, opening subsequent apps(since we are using setShape) make Chrome App unresponsive.
Please fix it asap as it is impacting our customers 

Reopening this, since the fix didn't solve the issue.

I tried the steps described in comment #20 on Chrome 70.0.3538.76 and 71.0.3578.49 on a Google Pixelbook and everything worked as expected (no issue with painting other apps).

Can you confirm :
1. On what device are you testing?
2. Is the setshape app in comment 20 affected?
3. If the setshape app is not affected, how can I reproduce the problem?

Thanks!

Citrix comment- we could reproduce the issue even with a sample app.(using setShape API).

Please find below the steps to reproduce the same:
1. Install the sample app (attached)
2. Launch sample (setshape_new app) it will create 2 windows 
4. click setshape button 
5. Click on the green window and try to interact with Google search box loaded inside Webview.
Observe that page loaded inside webview is not active (as we try to type)

Deleted: setshape_new.zip 1.9 KB

setshape_new.zip 1.9 KB Download

Pri-0 bugs are critical regressions or serious emergencies, and this bug has not been updated in three days. Could you please provide an update, or adjust the priority to a more appropriate level if applicable?

If a fix is in active development, please set the status to Started.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

[Potential RBS. Flagging for TPM team to review]

Citrix comment- Reattaching another sample with modified repro steps.(using setShape API).

Please find below the steps to reproduce the same:
1. Install the sample app (attached)
2. Launch sample (setshape_repro app) it will create 2 windows 
4. click setshape button 
5. Click on the green window . Dont immediately type in search box but click around on webview window and then  try to interact with Google search box loaded inside Webview. 
Observe that page loaded inside webview is not active (as we try to type)

Deleted: setshape_repro.zip 1.9 KB

setshape_repro.zip 1.9 KB Download

Also attaching video as to how we are seeing this issue

Deleted: Nov 30, 2018 3_03 PM.webm 1.7 MB

	Nov 30, 2018 3_03 PM.webm 1.7 MB View Download

We are observing this issue on  71 Beta and even 72 Dev channel consistently on irrespective of the chrome device.
Please help us fix this for our customers. Thanks!

I am unfortunately not able to reproduce on a Google Pixelbook running 71.0.3578.71 (beta). Can you paste the full content of chrome://version, or, if you prefer to share less information, provide the version number and device name?

If this was in 67, we cannot block stable on it as per comment 9 and 57

Note that in the worst case, we can just disable WebContentsOcclusion via Finch before M71 stable release. However, I would like to repro the issue before shipping that Finch config.

I also tested the setshape_repro, following the steps in the video, on a Lenovo Thinkpad Chromebook running 71.0.3578.49, and I was not able to reproduce the unresponsiveness of the webview. Everything worked fine for me. 
Citrix: Are you able to reproduce this on other devices? Could it be device-specific? 
fdoray@: Can you think of any other reason they and their customers would be seeing this issue but we cannot reproduce it? 

Shaped windows are not supposed to be considered when computing occlusion states, i.e. putting a shaped window on top of another window shouldn't affect its occlusion state (code: https://cs.chromium.org/chromium/src/ui/aura/window_occlusion_tracker.cc?l=293&rcl=611b929334504e26de879b7ea240162ff06fd0b0).

sky@: Are we experimenting with the window service on Chrome OS? I wonder if there could be occlusion tracking bugs when window service is enabled.

anishathattai@gmail.com: Can you provide the full content of chrome://version for the machine on which you are experiencing this issue? It would help us to know that experiments are active on this machine.

The WindowService is used for a couple of things on ChromeOS. The keyboard shortcut viewer, and if touch hud is enabled. I am aware of one bug related to this: 908629.

Attaching full content of chrome://version 

Deleted: Screenshot 2018-12-06 at 2.22.35 PM.png 206 KB

	Screenshot 2018-12-06 at 2.22.35 PM.png 206 KB View Download

Also just to add we have observed it on Chrome 71 Beta also 
Chrome 71.0.3578.49 Acer C720

The issue happens when we click on setshape button , click few times on secondary window and then try to access Google search box loaded inside Webview.

Pri-0 bugs are critical regressions or serious emergencies, and this bug has not been updated in three days. Could you please provide an update, or adjust the priority to a more appropriate level if applicable?

If a fix is in active development, please set the status to Started.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Could you paste the content of chrome://version, including field trial hashes? That way we can determine if a field trial is causing this. Thanks.

Citrix Comment:
Adding content from chrome://version

Please take this on high priority as we are seeing issues with this and this will hugely impact our customers.

Deleted: ChromeVersion.docx 7.8 KB

ChromeVersion.docx 7.8 KB Download

Citrix Customer comment:

When 1 Citrix app is launched it siezes control and no more Citrix apps can be launched because the hand tool doesn't appear when I mouse over other apps. Also, scroll doesn't work on a pdf open in a browser window, Excel online mouse doesn't work properly. All functionality returns when the first Citrix app is closed.

 

 
About my device : Lenovo N22 running Chrome OS Version 71.0.3578.94 (Official Build) beta (64-bit)

Device model : <model of the ChromeBook> 




Citrix Workspace app version : 18.11.3.4057
Useragent : Mozilla/5.0 (X11; CrOS x86_64 11151.59.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.94 Safari/537.36
 

Pls fix this issue with 71 stable channel, 
Customer comment: 
I am managing over 20,000 Chromebooks in our domain. We have a few devices on the Beta Channel for testing in order to spot potential problems before they reach the Stable Channel.
 
This issue is common in Beta releases and cripples my Chromebook so badly it is difficult for me to test the Beta release properly. I have encountered it in versions 68.0.3440.40, 69.0.3497.35, and now 71.0.3578.94. 
 
What needs to happen so that this issue is resolved in future Beta releases?

1. Have you encountered the issue on any 70.* version?
2. Are there 71.* versions with which you did not experience the issue?
2. Have you encountered the issue with the 71.* version that is currently being rolled out to the stable channel?
3. Can you provide detailed steps to reproduce the issue with the Citrix app? (how to install the app, how to reproduce the issue, what is the observed and desired behavior, test account if required)? Unfortunately, I'm not able to reproduce the issue on 71.* with the sample apps provided on this bug.
4. If you navigate to chrome://discards in Chrome and partially occlude the Chrome window with a Citrix shaped window, what value do you observe in the "visibility" column for chrome://discards?

I'm investigating what can be done to fix this issue.

A few more questions, in addition to the questions above:

5. Do you have the issue with the sample apps in comments 69 and 73, or only with one of these sample apps? Is the issue that any window under a shaped window is not interactive, or that only a <webview> under a shaped window is not interactive?
6. Do you experience the issue with version 72.0.3626.30+ on dev channel?
7. Does disabling "draw occlusion" in chrome://flags fixes the issue?
8. Does disabling "Out-of-process system UI (mash)" in chrome://flags fixes the issue?
9. Does disabling "Cross process frames for guests" in chrome://flags fixes the issue?
10. When a window is not interactive, are you able to make it interactive by selecting it in overview mode? (access overview mode with this key https://kbimg.dell.com/library/KB/DELL_ORGANIZATIONAL_GROUPS/DELL_GLOBAL/Content%20Team/keyboardnexttab_Chromebook11_BK.jpg on the Chrome OS device keyboard).
11. When a window is not interactive, are you able to make it interactive by selecting it in Alt-Tab view?

Here is our reply to few questions asked. We will provide answer to remaining questions asap.
1. Have you encountered the issue on any 70.* version?
    [Citrix]:  We have not seen this issue on 70.* stable channel.

2. Are there 71.* versions with which you did not experience the issue? Have you encountered the issue with the 71.* version that is currently being rolled out to the stable channel?
  [Citrix] :     With stable 71.*, we have not seen the issue.

3. Can you provide detailed steps to reproduce the issue with the Citrix app? (how to install the app, how to reproduce the issue, what is the observed and desired behavior, test account if required)? Unfortunately, I'm not able to reproduce the issue on 71.* with the sample apps provided on this bug.
 [Citrix]:  Did you try to reproduce on beta 71.0.3578.49?

5. Do you have the issue with the sample apps in comments 69 and 73, or only with one of these sample apps? Is the issue that any window under a shaped window is not interactive, or that only a <webview> under a shaped window is not interactive?

[Citrix] : Issue is reproducible with sample app in comment 73. We have seen this issue with <webview> under a shaped window.

6. Do you experience the issue with version 72.0.3626.30+ on dev channel?
[Citrix] : Issue is reproducible with 72.0.3626.30 dev channel.

[Citrix] : Following tests are performed on 72.0.3626.30 dev channel.

If you navigate to chrome://discards in Chrome and partially occlude the Chrome window with a Citrix shaped window, what value do you observe in the "visibility" column for chrome://discards?
[Citrix] : visible

7. Does disabling "draw occlusion" in chrome://flags fixes the issue?
[Citrix] : No, it didn't work.

8. Does disabling "Out-of-process system UI (mash)" in chrome://flags fixes the issue?
[Citrix] : No, it didn't work.

9. Does disabling "Cross process frames for guests" in chrome://flags fixes the issue?
[Citrix] : We couldn't find the setting in chrome://flags.

10. When a window is not interactive, are you able to make it interactive by selecting it in overview mode? (access overview mode with this key https://kbimg.dell.com/library/KB/DELL_ORGANIZATIONAL_GROUPS/DELL_GLOBAL/Content%20Team/keyboardnexttab_Chromebook11_BK.jpg on the Chrome OS device keyboard).
[Citrix] : No, it didn't work.


11. When a window is not interactive, are you able to make it interactive by selecting it in Alt-Tab view?
[Citrix] : No, it didn't work.

If I understand correctly, stable channel is not affected by this issue. Therefore, I don't think there is a need to ship a field trial configuration to disable WebContents occlusion on stable channel. I will investigate why you encountered this issue on Beta channel when I'm back from vacation on January 7th. I really want to avoid seeing this issue in future Beta releases.

Ping, also downgrading since this isn't an emergency as it's not affecting stable channel users.

Comment from Citrix:

Few customers are reporting this issue even on stable version just this week.
Their Chrome Details:

1) Device model : Dell Chromebook 11 3189, Chrome OS Version 71.0.3578.94 (Official Build) (64-bit)

Citrix Workspace app version : 18.11.3.4057
Useragent : Mozilla/5.0 (X11; CrOS x86_64 11151.59.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.94 Safari/537.36

2)About my device : 

Device model : Google Pixelbook


Citrix Workspace app version : 18.11.3.4057
Useragent : Mozilla/5.0 (X11; CrOS x86_64 11151.61.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36

Please look into it asap to help us.

Thanks!

Hi,

We're getting reports of this issue affecting our users. They are only able to launch a single application at a time. 

We have a device running 70.0.3538.58 that isn't affected by the issue so it looks like this is down to an upgrade to ChromeOS. 

This is a huge problem for us as a large Group of our users are now reliant on Citrix and ChromeOS and now unable to work efficiently. 

I was able to reproduce the problem described in comment 73 with Chrome Stable 71.0.3578.98 on a Google Pixelbook when the #enable-viz-hit-test-draw-quad flag is enabled in chrome://flags. The problem doesn't happen when the #enable-viz-hit-test-draw-quad flag is disabled.

Can someone confirm that the #enable-viz-hit-test-draw-quad flag is what causes the issue in the Citrix app?

ccing riajiang@chromium.org who changed the Finch config for this feature recently.

Hi,

I've just disabled chrome://flag viz-hit-test-draw-quad as requested and then rebooted my 2015 Pixel running 71.0.3578.94 and that did seem to resolve the issue. I can confirm that I am now able to launch multiple applications. 

I can confirm that I was able to repro the sample app problem in c#73 with Beta 71.0.3578.85 and disabling VizHitTestDrawQuad fixed the problem. I'll disable finch on Beta and Stable and look into fixing this.

Hi riajiang@chromium.org, do we have an ETA for a fix regarding this issue? We have a VIP customer that is being affected by this issue. Thanks!  

The finch experiment has been disabled on ChromeOS for beta and stable channel so this issue shouldn't happen anymore, could you check if you can still repro it? Thanks!

Citrix Comment: we are still able to reproduce the issue with beta build 72.0.3626.49. Kindly let us know if this is not the latest beta build and also pls provide the version number of beta and stable build where the finch experiment is disabled.

The issue should have been fixed for all versions and channels. Can you copy-paste the full version and "Variations" hashes from chrome://version on the device where you reproduce the issue? Can you also try to "Reset all to default" in chrome://flags?

Citrix Comment:

version : 72.0.3626.49 (Official Build) beta (64-bit)
Platform: 11316.66.0 (Official Build) beta-channel peppy

Variations: 
411b6d4e-f23d1dea
d01ab0d3-f23d1dea
73ebcf26-c982d8da
1a0d11d4-2f9febdf
cb1942f7-37e2eff0
2b6ab552-ca7d8d80
66df3e9d-5f3c1a6f
b7e2524c-1181467f
411c711e-3f4a17df
453f20e2-ee63ae02
38eb801c-3f4a17df
7c1bc906-f55a7974
9def365c-3f4a17df
2342e907-f23d1dea
47e5d3db-3d47f4f4
d442dfb7-ca7d8d80
71ed337-3f4a17df
81a62f29-48b9f718
a582a1b8-ad75ce17
e27776-48b9f718
8ee5ed19-ca7d8d80
74658432-ca7d8d80
6ab6f98e-ee63ae02
8e66b915-d3d69a93
116c6887-86d858e7
36115667-f23d1dea
55a3035a-f23d1dea
edbcf7c5-ddf1844d
5485fc4d-377be55a
93731dca-60872458
87a01a8d-ca7d8d80
41f007f9-f23d1dea
9b4c4257-6ad6e56e
165e16d1-3f4a17df
9e5c75f1-e10fa620
ed40b571-ae9eb0f3
2594bdf4-3e4b89c4
913f596c-48b9f718
2fccc5d5-3f4a17df
d1cd70a5-3f4a17df
b7e5dbe8-2bd20c23
4ea303a6-c2ab5f34
95876445-ca7d8d80
3f2db1c3-2ac6bc24
b363a81f-ba937583
67246da1-377be55a
cc54eb06-720b026c
58a025e3-36e97b2c
d4d220f9-1e107d07
4932440-f23d1dea
26464629-ee63ae02
51af0496-f23d1dea
51b9b54d-f23d1dea
7345ea6-ed937440
4bc337ce-377be55a
553edbc3-3f4a17df
d1466cda-983ee3
494d8760-52325d43
3ac60855-486e2a9c
f296190c-f9f7acb5
4442aae2-a5822863
ed1d377-e1cc0f14
75f0f0a0-d7f6b13c
e2b18481-5c63917a
e7e71889-e1cc0f14
f9e5da91-508355f5
d91d40b-ca7d8d80
85963571-ee63ae02
a4f1de9f-f23d1dea
82ebe475-3f4a17df
cc73f8a1-ca02b375
b4e8892d-a3a14831
81c6897f-e872b86f

riajiang@: The VizHitTestDrawQuad is enabled by default in code since https://chromium-review.googlesource.com/c/1382751. It is not enough to end the Finch config that enables it, we need a Finch config that explicitly *disables* it.

That CL is landed after M72 branch cut so it shouldn't affect M72 beta tho

Actually, the original enabling by default was before 72 branch cut and the revert was after branch cut (same with the re-enabling). I'll land a finch config to disable it, thanks fdoray@!

The finch config to explicitly disable Viz hit-testing has been landed. 
Fixes to this bug with Viz hit-testing are underway but will most likely be in M73.

riajiang@
Thanks for all your work.
One of the Enterprise customer is affected this issue and wishing to solve by CrOS 72 instead of 73.
Also the deployment of the flag as a work around for several users is not feasible as well.
Hence, it would be grateful if you could release it as soon as possible.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/680301378b84b5808d02700ca46ac9bbd3835584

commit 680301378b84b5808d02700ca46ac9bbd3835584
Author: Ria Jiang <riajiang@chromium.org>
Date: Fri Jan 18 17:49:31 2019

Update hit-test data sequence to be aligned with RootRenderPass.

Instead of going over the render_pass_list in the CompositorFrame
and add in hit-test data from SurfaceDrawQuad along the way, this
CL changes it to follow the sequence referenced in the
RootRenderPass, e.g., a render_pass that's drawn on a temporary
texture first early in the render_pass_list can be referenced in
the RootRenderPass later than some other content.

viz_unittests

Bug: 864508
Test: site_per_process_hit_test_browsertests, cc_unittests,
Change-Id: I1a978cc0290ed5aa74d0dd95ad173256bace862e
Reviewed-on: https://chromium-review.googlesource.com/c/1414074
Reviewed-by: Sadrul Chowdhury <sadrul@chromium.org>
Reviewed-by: weiliangc <weiliangc@chromium.org>
Commit-Queue: Ria Jiang <riajiang@chromium.org>
Cr-Commit-Position: refs/heads/master@{#624188}
[modify] https://crrev.com/680301378b84b5808d02700ca46ac9bbd3835584/cc/mojo_embedder/async_layer_tree_frame_sink_unittest.cc
[modify] https://crrev.com/680301378b84b5808d02700ca46ac9bbd3835584/components/viz/client/hit_test_data_provider_draw_quad.cc
[modify] https://crrev.com/680301378b84b5808d02700ca46ac9bbd3835584/components/viz/client/hit_test_data_provider_draw_quad_unittest.cc
[modify] https://crrev.com/680301378b84b5808d02700ca46ac9bbd3835584/components/viz/common/BUILD.gn
[add] https://crrev.com/680301378b84b5808d02700ca46ac9bbd3835584/components/viz/common/hit_test/hit_test_data_builder.cc
[add] https://crrev.com/680301378b84b5808d02700ca46ac9bbd3835584/components/viz/common/hit_test/hit_test_data_builder.h
[modify] https://crrev.com/680301378b84b5808d02700ca46ac9bbd3835584/components/viz/service/frame_sinks/direct_layer_tree_frame_sink.cc
[modify] https://crrev.com/680301378b84b5808d02700ca46ac9bbd3835584/components/viz/service/frame_sinks/direct_layer_tree_frame_sink_unittest.cc