New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 724930 link

Starred by 13 users

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2017
Cc:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug



Sign in to add a comment

Chrome M58 fails to open any URLs (potentially due to compatibility issue with a third party app/tool)

Project Member Reported by kotah@chromium.org, May 22 2017

Issue description

Chrome version: 58.0.3029.96
OS version: Windows 7

We have multiple reports from enterprise users that Chrome fails to start since updating to M58. Symptoms are:

- Opening Chrome browser shows black screen on startup
- Opening chrome:// URLs shows black screen or "Aw, snap" error (so users can't check Crash IDs)

Screenshots: https://drive.google.com/open?id=0B3B7SKCSXU4SX1NWa2Z4b0loWjA

Issue persists after disabling antivirus softwares.

I also found a few crbugs reporting similar issues:  crbug.com/721358 ,  crbug.com/717929 ,  crbug.com/719283 
(I see we are asking reporters for crash IDs in all these bugs...but it's impossible as chrome://crahes crashes Chrome)

We are waiting for Chrome browser debug logs (https://www.chromium.org/for-testers/enable-logging) from affected users, but opening this bug now for tracking purposes. Let us know if there is any other info that would be helpful in investigation.
 

Comment 1 by kotah@chromium.org, May 22 2017

Steps to repro (though I can't repro locally):
1. Download Chrome installer of the latest version
2. Install and start

Comment 2 by kotah@chromium.org, May 22 2017

Owner: atwilson@chromium.org
@atwilson, do you have any advice on what info/logs would be useful in investigating Chrome crashes when chrome://crashes is not available?
Cc: atwilson@chromium.org georgesak@chromium.org
Owner: pastarmovj@chromium.org
Julian, any tips for helping debug issues like this? If the screen is black, it sounds like it's some kind of rendering pipeline issue (maybe GPU?)
First we need to understand the envrionment a little better.

1. Do these users run Chrome locally on the desktop or through some remote session Citrix, Terminal Services, VMWare etc.?
2. Are they running the 32 or 64 bit version of the OS and Chrome (check in chrome://version).
3. Does running Chrome with --no-sandbox fix the issue. (Don't propose this as a solution ever. This is just to verify whether the sandbox is the issues).

Let's see what is in the logs and the answers to this questions and we can try to narrow down the issue with further steps. 

Comment 5 by roy...@google.com, May 22 2017

Labels: M-58
Cc: vkhabarov@google.com
So what I've seen - 
1. Not sure, some are on VMWare
2. All Win 7 64 bit, Chrome - 32 or 64 bit
3. --no-sandbox helps in all cases so far.
Cc: wfh@chromium.org
Ok let's see what will show up in the logs once you have some.

So far from the results in 3 it is something related to the sandbox in Chrome but since we have very little other info we can't say much about what exactly.

One follow up question to your answer to nr.2 when you say Chrome is 32 or 64 bit does it mean both exist and fail? We have had some issues only appear with the 64 bit version that is why I am asking.

Adding wfh to the cc list.

Comment 8 by wfh@chromium.org, May 23 2017

Labels: Needs-Feedback
please try running chrome with command line:

--enable-logging=stderr --v=1 --vmodule=metrics=2 > log.txt 2>&1

then attach log.txt to this bug. If customer unwilling to attach entire log (it contains no PII) then just paste the result of the Process.Sandbox.Launch.Error histogram. Thanks.
I faced the same issue.
I restarted my windows, without closing the chrome. After restart I opened the chrome, it asked to restore the old session, i clicked restore after that all tabs are showing black screen only. In the background web url is correctly opened as on hover it is showing the ad urls
Capture.PNG
29.0 KB View Download

Comment 11 by wfh@chromium.org, May 24 2017

crashes in #10 are from 15 may, are they recent? has the customer been able to use the extra debugging logging command line in #8 to give more details?
wfh@ - crashes from #10 are the latest we have from customer.
Here is the link to debug log requested in #8: https://drive.google.com/drive/folders/0By9xEkAUmm_cYi1YTnhNc1AxNTg?usp=sharing
Any Updates?? Is the issue resolved?

Comment 14 by roy...@google.com, May 31 2017

Labels: -Needs-Feedback
We should have the data requested in #8. Whats the next step ?
One of my customer provided logs from yesterday. Here is the link https://drive.google.com/open?id=0B51czt8vCRpUR2h3VXNUYmxPblU

This requieres Google Credentials. Please let me know if you have any updates or if you need any further information.
Another log from customer - https://drive.google.com/open?id=0B7RXwPjBEiJ3Sk5KUGRLWFc2RlU
All affected devices are physical in this case.

Histogram: Process.Sandbox.Launch.Error recorded 5 samples (flags = 0x41)
5   ------------------------------------------------------------------------O (4 = 80.0%)
31  ------------------O                                                       (1 = 20.0%)


One of our customer provided logs from 6/2/2017 https://drive.google.com/drive/folders/0B51czt8vCRpUVl9ERFU5U0RDUXc?usp=sharing

If they use --no-sandbox if fixes the issue but customer wants to know a root cause and when will this be fixed. 

I would appreciate your input

Comment 19 by ysha...@gmail.com, Jun 8 2017

Awaiting Resolution. Please resolve at the earliest.
we have some crashes from a customer:
Chrome 58.0.3029.96 (64-bit)
physical PCs
Windows 7 64 bit.
crash ID:
6d144085f0000000
69a2caae40000000
42aae6f198000000
66522085f0000000
b3ea584e40000000

Comment 21 by jayhlee@google.com, Jun 21 2017

pastarmovj@ what's status on this?
Cc: marcore@google.com
Labels: M-59
we have another customer on a citrix system.
if they use --no-sandbox it works
Chrome version found 59.0.3071.104
I've asked for the console log

Comment 23 by kotah@chromium.org, Jun 22 2017

Labels: -M-59
#c20 - If user can open chrome://crasehs and check Crash IDs, then it's a different issue. Those crash IDs indicate crbug.com/660291.

#c22 - That sounds more like  crbug.com/699043 . Try https://support.google.com/chrome/a/answer/7380899
Hello Guys,

Customer has tested using --No -Sandbox option and issue is fixed, however, she would like to have a RCA and when it will be fixed. Customer was kind enough to capture Crashes ID, Policy Page, Extensions, Logs, Debug Logs. Here is the information https://drive.google.com/drive/folders/0B51czt8vCRpUY0toQzZRbU9sX00?usp=sharing

I appreciate your help

Comment 25 by ysha...@gmail.com, Jul 3 2017

Any update? when this problem will get resolved?? Pls do this on highest priority...
Cc: blumberg@chromium.org
Hi, 

We also had one customer who experienced this issue, it is now working again.

It seems that the problem was about the virtualization. They were running Chrome 64bit on Windows 7 64bit Enterprise Edition virtual machines running on VMware cluster, ESXi version 5.5. With ESXi 5.5, the only way to run Chrome 64bit was with --no-sandbox option. Without it no matter what you do you get Aw, snap. I could not enable crashes, I could not open a new tab, I could not open the option menu.

Thay decided to upgrade the infrastructure to the latest VMware version ESXi 6.5 and together with it we updated the VMware agents installed on all Win 7 virtual machines. Then they tried to run Chrome again without --no-sandbox and it is now working.

Hope this helps. 
Hello guys. Is there any updates for this issue?

Comment 29 by ysha...@gmail.com, Jul 26 2017

awaiting revert since last 3 months.. pls resolve on highest priority
Status: Assigned (was: Untriaged)
Hello please excuse the long radio silence on this bug. It got entangled in another ongoing investigation but I believe it is a separate issue and would like to move on with the investigation here.

While investigating the crash dumps listed in c#20 I notice that there is some third party software called "Blue Coat Proxy" which seems to be now a Symantec product maybe called "Symantec Secure Web Gateway: ProxySG & ASG" that is injecting itself in the Chrome processes. Given that the crash is happening in initializing our security mechanisms it could be some compatibility issue. Can you try disabling/uninstalling this software on one affected machine and verify if Chrome runs fine without it. If not please collect a crash report without this software running and provide the crash ids for further investigation. If this turns out to be the problem you should try to see in the documentation how to prevent the dll injection of its dll in the Chrome process or contact Symantec for further assistance.

Comment 31 by roy...@google.com, Jul 31 2017

Summary: Chrome M58 fails to open any URLs (potentially due to compatibility issue with a third party app/tool) (was: Chrome M58 fails to open any URLs)

Comment 33 by roy...@google.com, Jul 31 2017

Igor: 
- Can we request customer to isolate what change bluecoat is making to the traffic ?
- Is it just doing SSL inspection, or is it doing content injection as well ?
- Being able to diff headers or content would significantly speed up the root cause of crash on our side.


Cc: scottmg@chromium.org
+scottmg FYI and in case you can provide advice what is this software doing that it shouldn't be doing. 

Scott, Will, I presume there is nothing we can do in Chrome that will not weaken its sandbox that will fix the problem? It is rather an issue in Blue Coat that they are being over-intrusive in the way they instrument other processes,
Can we get a local repro with this software?

I see it uses EasyHook64 per the crash. This is probably https://easyhook.github.io/.

We currently block only EasyHook32.dll
https://cs.chromium.org/chromium/src/content/common/sandbox_win.cc?rcl=beb884227d7b8759c461fc6ab43effb8ad479e2e&l=79 which seems fairly arbitrary. We should consider blocking easyhook64 as well, but it's of course hard to know if other hookers might cause instability then. (Presumably not too much if we're already blocking the 32 one?)

Blue Coat not-doing-that would be the best solution of course.
Scott, let's make a canary build with EasyHook64.dll blocked as well and ask the reporter to verify if this fixes the issue for them. WDYT? 

Meanwhile we will see if there is a chance to get a local setup to test this environment. 
Yeah, sgtm.
Project Member

Comment 38 by bugdroid1@chromium.org, Aug 10 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4a872b8f0bf932724137debf6aebdde521634dfa

commit 4a872b8f0bf932724137debf6aebdde521634dfa
Author: Julian Pastarmov <pastarmovj@chromium.org>
Date: Thu Aug 10 07:24:45 2017

Add easyhook64.dll to the exclusion list kTroublesomeDlls.

This DLL is used by BlueCoat and causes Chrome to crash on Windows 7.

BUG= 724930 

Change-Id: I4c2376f55a7d116d34f968287fe3c1eb3d4dbbde
Reviewed-on: https://chromium-review.googlesource.com/602267
Reviewed-by: Scott Graham <scottmg@chromium.org>
Reviewed-by: Will Harris <wfh@chromium.org>
Commit-Queue: Julian Pastarmov <pastarmovj@chromium.org>
Cr-Commit-Position: refs/heads/master@{#493318}
[modify] https://crrev.com/4a872b8f0bf932724137debf6aebdde521634dfa/content/common/sandbox_win.cc

Comment 39 by ysha...@gmail.com, Aug 10 2017

Can someone please take access to my PC & resolve the issue? I am waiting since last some many months to get this resolved. Pls Call +919833039099 to take access & resolve
Hello yshah,

thank you for your cooperation. As you can see from c38# a speculative fix for this issue has just been introduced in the browser. I expect to have a canary release with this fix in the next 24h and once this is fact I will ask you to try to install and try the canary version of the browser and see if the problem is fixed for you. 
Stay tuned for new post here once the release is present.


Friendly Ping!

yshah have you had a chance to verify recent canary/dev versions of Chrome?


Comment 42 by roy...@google.com, Aug 29 2017

Julian: One of the customers confirmed this works. I've asked another case owner to check on this as well.
Issue 660291 has been merged into this issue.
Status: Fixed (was: Assigned)
This has been fixed for a while already. I am marking it fixed. Please ping back if this is wrong.

Sign in to add a comment