Well my Win 10 Enterprise just updated to 1803 a few days ago and I got the same CryptSvc timeout issue.
Upon reading the comments I simply deleted the;

HKEY_CURRENT_USER\Software\Microsoft\SystemCertificates\Root

key (exporting first of course) and all instantly started working again.
All the keys under this root were recreated.
I did restart the CryptSvc but not sure if I had to or not.

I think I have found a decent workaround for this.
I created a script that I pushed through SCCM to machines on 1803 to force a batch file that stops cryptsvc, changes the service account and then starts.

So far seems to be doing okay.

sfor@ Look here 7/13-7/28 for Ole Pilman, Clouver, PhiloMatch replies about subsequent problems after changing the CryptSvc logon account.  These may not apply to you or maybe you found a less aggressive approach.

https://productforums.google.com/forum/#!topic/chrome/s5S1uPI0kMc

Good luck! Keep us informed..

We learned that most, if not all, of our problems resulted when we switched
from local to cloud filtering with our Meraki firewall appliance. The
connections timeout waiting for the Meraki filtering process.

sfor@ c#112 - Did the CryptSvc login account change work?  Which account did you use? Local system?

It works, but unfortunately it has all sorts of side effects. Not a good solution at all. I have tried multiple accounts. The one that seems to have the smallest side effect is the domain administrator.

Honestly, at this point I'm really frustrated. This has been a problem for 4 months now with no resolution from anyone. (Or even an official recognition from anyone that it is a problem)

We determined that our Meraki MX400 firewall content filter was scanning
the full domain list and was timing out. Apparently the full list had grown
to large. We switched back to just screening the top domain list and voila!


*Glenn Dahlen*


*Business Officer*
750 E. 9th Street | Charlotte, NC 28202
<https://maps.google.com/?q=750+E.+9th+Street+%7C+Charlotte,+NC+28202&entry=gmail&source=g>
T 980.207.5889   F 704.342.4115

www.TEScharlotte.org <http://www.tescharlotte.org/>


*Creating scholars, nurturing spirituality **and embracing diversity in
Charlotte's center city.*

For us it is also custom certificates that is causing the problem, but I can't live without them at this time.

sfor@ - Did you try going through GSuite for Education support, to get attention for this CR and the lab PCs?  It's a stretch, but Google at least likes to highlight the classroom support.  With the right eyes they can probably identify where the permissions conflict is.

It seems we are experiencing this issue as well. We updated our machines to 1803 over the summer. And now some of them are having this issue. And touching each machine is not a good option. What are the side effects of just stopping the cryptsvc and not allowing it to run?

warr@ To verify you have 'this' cryptSvc problem, try
  starting IE or Edge before Chrome
  use a different Windows login account
If either/both of these avoid the problem, you likely have this cryptSvc problem.

The only known workarounds are the registry update or using a different Windows login account.

Disabling (not stopping) cryptSvc will likely interfere with other security checks.  I believe it is used in all program ID checks. Look here
http://servicedefaults.com/10/cryptsvc/
The text description is lifted from Microsoft, but I don't have the link.

If you are an educational institution and have GSuite for Education, try leveraging the GSuite support to get some attention here.

I have this bug as well. 

I have replaced my Registry permissions now and am waiting to see if it recurs, however I wanted to make a note on a comment I saw in regards to root cause. 

Comment 86 "When Chrome (through CryptSvc) attempts to interact with those registry values, CryptSvc is unhappy with the permissions and gets stuck in an infinite loop trying to correct it."

This is likely incorrect as Mozilla FireFox, Microsoft Edge, and Microsoft Internet Explorer are unaffected by this. If CryptSvc actually had a problem with the permissions alone, then these other products would also be affected. There must be another interaction between Chrome and CryptSvc specifically that contributes to this issue. 

> This is likely incorrect as Mozilla FireFox,

Firefox does not use CryptSvc, or CryptoAPI for that matter. It ships its own hermetic stack.

rsleevi@ kkalur@ - Since (still) unconfirmed this failure vector is getting no attention, although still a problem for affected users.

The registry repair works, but is not suitable for vanilla consumers.  There is something unique about the way that Chrome uses Protected Roots that Edge doesn't.  Too bad this is not going to be addressed, but at least there is a workaround.

Institutional users are another issue.  The registry repair is for HKCU, institutions need an HKLM patch or similar.  It is generally not possible to apply the HKCU patch for every users in the domain.

sfor@, warr@ Can you post your problem, with back ref to here, in chrome-admins?
  https://productforums.google.com/forum/m/#!topic/chrome-admins/iYPdnrr5HtM
Maybe they will have a better suggestion.  Please confirm with a chrome-admins link.

sfor@, warr@ were you able to get any support from the G-Suite community?

There is still experimentation with changing the cryptSvc login account to Local System account.  Adding "Allow service to interact with desktop" allowed mmc.exe, services.msc, etc, when not logged with admin permission.
Mentioned here: 
https://productforums.google.com/d/msg/chrome/SYBB6iGbnfQ/Ctd8mmb4AgAJ

I had not pursued the local system signin because of the things it broke. If adding desktop access fixes most problems, I'll add it to the workarounds. I need more feedback before moving on.

There was also another mention of changing the .exe name to chrome..exe (Frederic 24-Aug), but I suspect this reflects an install/update problem, not the cryptSvc conflict at fault here.  Using chrome..exe also has interoperability issues.
Mentioned here
https://productforums.google.com/d/msg/chrome/s5S1uPI0kMc/5RHZSijqAQAJ

Comments appreciated, particularly on the cryptSvc login change.

I actually gave up and decided to go back to LTSB 1607. I recreated my master image and am slowly reimaging every machine as they decide to update to 1803 and this problem rears its ugly head. 

We switched the log on account from Network Service to a domain account. It seems to have fix the issue. We will continue to test this workaround. Still had to touch each computer. Could not make the change via GPO.

After doing Network service to a domain account, i cant use windows update anymore. after a restart.

Correct. That is why I abandoned the idea of replacing the service account as a solution and decided to just go to LTSB 1607 and not have to worry about this anymore.

ok, is there a way to block the windows update 1803 from downloading,i wil rolling back to the 1607 to, i hope there wil be a fix soon.

I am still facing this issue with tens of windows users in my org. Read through a ton of documentation, but it is not clear to me yet. In one of the vm that I was using for testing after downgrading it to Windows 10 1803.17134.112 build, this problem went away after a few days. Did Chrome rollout a fix for this?

I recently reinstalled my windows 10 laptop with 1803 and can confirm it also happens to me on 69.0.3497.81 stable

I was able to make a batch script to fix the registry using SetACL.  Now if only we can find a way to detect if the key has the permission order issue, so we can run it through powershell and only fix if the error exists.
----------------------------------------------------
@echo on
net stop cryptsvc

SetACL.exe -on "HKEY_CURRENT_USER\Software\Microsoft\SystemCertificates\Root" -ot reg -actn setowner -ownr "n:Administrators"
SetACL.exe -on "HKEY_CURRENT_USER\Software\Microsoft\SystemCertificates\Root\ProtectedRoots" -ot reg -actn setowner -ownr "n:Administrators"
SetACL.exe -on "HKEY_CURRENT_USER\Software\Microsoft\SystemCertificates\Root" -ot reg -actn ace -ace "n:Administrators;p:full"
SetACL.exe -on "HKEY_CURRENT_USER\Software\Microsoft\SystemCertificates\Root\ProtectedRoots" -ot reg -actn ace -ace "n:Administrators;p:full"

reg delete HKEY_CURRENT_USER\Software\Microsoft\SystemCertificates\Root\ProtectedRoots /f
reg delete HKEY_CURRENT_USER\Software\Microsoft\SystemCertificates\Root /f

net start cryptsvc
exit
---------------------------------------------------

mhaa@ For personal accounts a direct registry repair is available here
https://productforums.google.com/forum/#!msg/chrome/s5S1uPI0kMc/PVBgVbx6DAAJ
and does not involve modifying the CryptSvc service or account.

The effect should be the same as znor's powershell commands.

Before you do the registry repair, either with the script or from regedit, if you could log your registry state and post the log file, this would be helpful.  From Windows PowerShell use this:

$Rootkey="HKCU:\Software\Microsoft\SystemCertificates\Root"
$PRkey=  "HKCU:\Software\Microsoft\SystemCertificates\Root\ProtectedRoots"
$logfi="C:\Temp\protroot.log"
Get-Acl -Path $Rootkey | Format-List >> $logfi
Get-Acl -Path $PRkey   | Format-List >> $logfi

Repeat after the repair, and post.  You can view and edit the log file with any text editor (WordPad, NotePad..)

znor@ Thanks.  I was working on the same powershell commands yesterday, after transitioning from the old subinacl.  I'm not sure about setowner as admin under HKCU.  Shouldn't these continue to be owned by the user with extended admin perms?  The original config is for ProtectedRoots to have full for CryptSvc and Read for user.

thx, Larry

I'd like to find a fix that generic users can use.
The registry repair is still a reach for some (see here)   
 https://productforums.google.com/forum/#!msg/chrome/s5S1uPI0kMc/PVBgVbx6DAAJ
Admin add-ons like SetACL.exe and subinacl, require users to load yet another tool, I'd like to avoid them if possible.
The Win10 PowerShell builtins for Set-ACL/Get-ACL should work OK.
I'm looking into that...

I bundled znor's SetACL commands into a PowerShell script (attached with log). You'll need to download the SetACL.exe admin tool from HelgeKlein here
  https://helgeklein.com/download/
You need to run the script from an admin powershell window in the folder where you installed SetACL.exe.  See the Forum instructions to enable powershell scripts.

The script also restores and HKCU certificates lost when deleting the Root key.

It generates a c:\temp\ProtRoots.log file documenting the initial ACL permissions.  Please post (a sanitized version) of the log, so I can see what the initial problem state was.

Full instructions are here in the Help Forum:
https://productforums.google.com/d/msg/chrome/s5S1uPI0kMc/b2Mud3SKCgAJ

Let me know how it goes..
If you have problems, attach the shell console output.
The ProtRoots.log only logs the ACL info and doesn't trap script output.

Deleted: ProtRootsFix-SetACL.ps1 2.4 KB

ProtRootsFix-SetACL.ps1 2.4 KB Download

Deleted: ProtRootsFix-SetACL.log 2.0 KB

ProtRootsFix-SetACL.log 2.0 KB View Download

So i ran the ps script succesfull, but no effect in chrome.
Still slow and getting err_timed_out
I Attached the log
winver = 17134.345

Deleted: ProtRoots.log 3.5 KB

ProtRoots.log 3.5 KB View Download

@niels: err_timed_out is probably a different (network related) problem.  Do you see any cryptSvc usage in task manager?  Look here for instructions:
https://productforums.google.com/forum/#!msg/chrome/s5S1uPI0kMc/PVBgVbx6DAAJ

If you see cryptSvc usage, I'll follow up later.

Hi Larry,

Its def the cryptsvc issue. cpu goes up to 10% and then i get the establish secure connection on the left and then sometimes the error_time_out.
I ran your ps1 but no fix.
The thing is when i check the ProtectedRoots rights before running, the user is already owner of ProtectedRoots.
When i change service logon to local account it works fine but thats not a permanent fix for us because of domain pc's.

So i found out when we do *NOT* use a mandatory profile in windows, chrome runs good.
Using a normal profile fixed our situation.

niels@ - The powershell script is only intended for personal accounts - sorry.  If you look back in the discussion here, you'll see that we don't have a solution for managed domain user accounts.  See sfor's comments.

Same problem with mandatory profiles also reported:
10/1 (very recent) Danny Field: same solution: use local instead https://productforums.google.com/d/msg/chrome/s5S1uPI0kMc/k_1WGrBTAwAJ

7/5 Ranolph - OK for managed roaming, only fails for mandatory
https://productforums.google.com/forum/#!topic/chrome/tHMs-rm4Pvs

The registry ProtectedRoots owner is not as important as the permissions -
CryptSvc needs to have full perms.

FYI: This article describes issues in setting up mandatory profiles.  
https://james-rankin.com/articles/creating-a-mandatory-profile-on-windows-10-1803/

If you make any progress on this let us know..

Same issue here - we use roaming profiles here with XenDesktop and cannot use Chrome because of the CryptSvc Problem - if I use chrome as a "normal" use it does'nt work - if I open Chrome on this account "As Administrator" it works...

Robert

off@ #144 - Roaming profiles are well supported for both Enterprise and Education Chrome.  Please contact either the Enterprise or GSuite for Education resources - or XenDesktop, as this may be an Xen config issue.

The Enterprise Admin forum is here:
https://productforums.google.com/forum/m/#!categories/chrome-admins

See attached screenshot of my healthy ProtectedRoots perms.  Note: no inheritance, and full control by CryptSvc.  Owner can be either System or User, it doesn't matter.

off@: If you find a solution, please let us know.

Deleted: CR838707_HKCU_4ProtRootsAdv.png 25.1 KB

	CR838707_HKCU_4ProtRootsAdv.png 25.1 KB View Download

 larrylac...@ - we have switched to Edge as the Standardbrowser until this problem is fixed

off@ #146 - If you look at the 7/5 Ranolph link above, they had no problem with roaming profiles.

In general I think roaming profiles are OK.  There's probably another domain or XenDesktop issue that needs to be resolved.

Win10 1809: The Win10 fall feature update 1809 was released for public (consumers) 10/9 as KB4464330 build 17763.55.  I saw one user mention that it solved his '1803 Slow Chrome startup' problem:
https://productforums.google.com/forum/#!topic/chrome/MOKEjJbPmKg

Maybe it will sort your domain roaming problem too..

larrylac...@ the solution to give the Cryptography Service Local System Account has many side effects => mmc.exe, Windows Update etc.
it is not save for us to change this for a master image for hundrest of people...

off@ #148: Agreed: changing the CryptSvc login account to Local System or Admin has many side effects.  

There was a secondary workaround ("Allow service to interact with desktop") that fixed somethings (*.msc, mmc), but probably not all (Windows updates..), so I don't recommend it.  

Sometimes I wish that the CR's were Wiki pages or Google Docs so we could build a comprehensive summary, instead of having to constantly rescan this (and several other threads).

znor@ #134: Re: SetACL.exe usage

There are two different use cases for the windows user account
a) user has administrator privs
b) user does not have admin privs

I reworked you #134 .bat example as a powershell script, run it in a powershell admin console, for a type-a user, and it works fine.

If I do the same for type-b (no admin privs), with a powershell admin console, the cmd
 set Rootkey=HKEY_CURRENT_USER\Software\Microsoft\SystemCertificates\Root
 set PRkey=%Rootkey%\ProtectedRoots
 set regPath=%PRkey%
 SetACL.exe -on %regPath% -ot reg -actn setowner -ownr "n:Administrators"
uses the admin ProtectedRoots, not the ProtectedRoots of the type-b user I'm trying to repair.

Suggestions? Maybe I only need to use PowerShell admin, if the user account is an admin account?  If so, how to detect admin privs in powershell?

TIA..


  

So I contacted Google G Suite support. As we feared they blame everything on Microsoft and ARE NOT going to fix this problem. I'm not sure why no one from Google or Chromium has been willing to say that on this post. But it is what it is.

Seems like we won't get this problem fixed ever. Microsoft will simply blame Google.

Here is their exact message

Thank you for replying, as mentioned in the description of this ticket and the comments of https://bugs.chromium.org/p/chromium/issues/detail?id=838707 the issue happens after an update in the OS. The engineering team has verified the problem and they confirmed that this will not be addressed by us, I wish I could provide a solution for this case but since the problem has already been confirmed as a problem with the specif build of Windows this should be addressed with them.

sfor@ #152: What kind of profiles do your AD Windows users have? Roaming? Local?

It's hard to believe the profile problem has not become a deal breaker with institutional, enterprise users.

FWIW: Early reports are the problem persists with 1809 RS#5.

Local profiles exclusively.

And yes, 1809 still has it.

I don't understand why this bug report still says unconfirmed when obviously it's an actual confirmed thing. They just won't fix it. 

(MarkSt (deleted #156,#157))

For the latest fix-it script look here
https://productforums.google.com/d/msg/chrome/SYBB6iGbnfQ/otXlCLgxBgAJ

which fixes the elevation conflict (see next comment)

(AFAIK) The SetACL tool in znor@ 9/12 #134 requires elevation to run.  When run in an admin/elevated environment, the HKCU refs point to the admin HKCU of the elevated environment, not the user HKCU.

The powershell script in 9/19 #137 did not account for this.  The attached script CryptSvcAdminSetAclFix.psl does.  See log for execution details.

Deleted: CryptSvcAdminSetAclFix.ps1 5.1 KB

CryptSvcAdminSetAclFix.ps1 5.1 KB Download

Deleted: CryptSvcAdminSetAclFix.ps1-ia29.log 3.6 KB

CryptSvcAdminSetAclFix.ps1-ia29.log 3.6 KB View Download

goanuj@, robertshield@, georgesak@: Downgrading this CR by removing labels 
-TE* is not going to bring any attention to this problem.  

While it is not a hardware problem and cannot be reproduced in the lab, there are technical users who can provide dumps and logs, just ask.

Problem can be solved by moving a user to a new and clean user account. The problem started when I installed Google Chrome on a fresh but outdated Win 10 Home (DVD ISO downloaded from MS website) and then updated to the latest Windows 10 release. Installing a recent Chrome on an old Windows 10 seems to screw up the Chrome user configs - if there is such a thing. The probably easiest workaround is to simply delete all Google Chrome user config data, though I haven't tested this yet directly.

Markst #163: Using an alternate Windows account is a known workaround, see the list at the bottom of this forum post:
https://productforums.google.com/forum/#!msg/chrome/s5S1uPI0kMc/PVBgVbx6DAAJ
The problem is linked to the windows account used to install windows and is noted often after a fresh reinstall and update cycle.
Swapping accounts is not always feasible.

The problem stems from bad CryptSvc registry permissions for the user - another type of config data.  Wiping/restoring the Chrome profile config data has no effect, it's inside the registry, tied to CryptSvc.

The fix-it script described here
https://productforums.google.com/d/msg/chrome/SYBB6iGbnfQ/otXlCLgxBgAJ
is focused on this specific registry problem and is the most direct approach.  It has the side-benefit of logging the original CryptSvc registry permissions (ProtRoot.log) which may help us pinpoint the problem.

 Issue 903872  has been merged into this issue.

Susan@, nhar@ - If someone will look at them, I can get users to supply net logs or dump files, to help find, repro this problem.

All: Please use the script & instructions from the forum to get the latest:
https://productforums.google.com/d/msg/chrome/SYBB6iGbnfQ/otXlCLgxBgAJ

The attachments above #159 are slightly dated

Could someone affected by this capture a ProcMon log? Instructions can be found here: https://blogs.msdn.microsoft.com/dswl/2010/01/10/how-to-capture-a-process-monitor-trace/

Make sure Chrome hits an ERR_TIMED_OUT while capturing.

It's possible the log will exceed the attachment size limit for the bug tracker. If that's the case you can place it somewhere on Google Drive or some other site and send us a link.

Hopefully a log should shed some light on what's going on.

Here you are. https://drive.google.com/file/d/1zerWGOr3exBLN3EQb6zMzS13Ew6-Wz7n/view?usp=sharing

Its an enormous file, but I wanted to send you everything. This way you can filter it how you want.

here you can download a ProcMon log:
https://drive.google.com/file/d/1udirgw6VqaURG8bV-ytiOoXMUXhtnuO9/view?usp=sharing

robert

Nov.19 Lee UBS reports CryptSvc logon workaround for AD mandatory login profiles:

..we are using Windows 10 Eng 64bit Enterprise 1803 or 1809 clients and Windows 2016 server AD with mandatory login profile, Chrome is V70 64bit default profile is roaming also.

What I did with GPO was 
 change the Crypto services logon from Network services 
  to local system account 
 allow services to interact with desktop 
 added domain users to the Crypto services to start/stop.

Chrome seems to work OK now.

See: https://productforums.google.com/d/msg/chrome/s5S1uPI0kMc/mib23iqtCAAJ
 

Thank you for providing more feedback. Adding the requester to the cc list.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

sfor@ other AD admins..

Hopefully we see some progress.. if not and you still need workarounds..

One of the problems with AD admin scripts is that HKCU resolves to HKLU of admin, not the HKCU of the user.  My powershell script makes a crufty attempt to find the HK_Users profile for the target user.  You might be able to use a similar approach.  PS provides some nice hooks for getting the SID.

Lee's change to the CryptSvc logon account with desktop access and domain users access to CryptSvc looks promising.

This old workaround is still surfacing:
Go Settings> Advanced> System> Proxy settings> LAN> disable auto detect
See: 11/20 https://productforums.google.com/d/msg/chrome/MOKEjJbPmKg/wKpiFtf1CAAJ

Hi Larry

Here is the process monitor file you asked for.

https://drive.google.com/file/d/1Gvx8qRnKnLrzkGtKQTN8KfoKOMT9t6aR/view?usp=sharing



Thanks

Re: AD procmon logs: These may not reflect the same problem home users have. This includes the sfor#169, off#170 and le#174 logs.

In the non-managed home environment, switching to an account that did not install windows generally avoids the problem.  I.e. the problem is tied to the home account that installed Windows.

In an AD environment, the problem only shows up on user accounts, which did not install windows.  Using a local admin account generally avoids the problem - which may have been associated with the Windows install.

And there's still the quirk, that starting(and holding open) IE or Edge before Chrome, allows Chrome to access CryptSvc OK.

@Larry "And there's still the quirk, that starting(and holding open) IE or Edge before Chrome, allows Chrome to access CryptSvc OK"

I cannot comfirm that - if I open IE and then chrome it works somtimes and sometimes not but if chrome is working and I made a refresh or access a page with IE then also chrome works (CryptSvc CPU goes down)

@off Did you mean: 
I cannot comfirm that - if I open IE and then chrome, it works sometimes and sometimes not, but if chrome is NOT working and I refresh or access a page with IE then chrome works (and CryptSvc CPU goes down)

Ich konne dass nicht bestaetigen - wenn ich IE aufmachen und dann Chrome, manchmal klappt es, manchmal nicht.  Aber wenn Chrome nicht funktioniert, und dann in IE ich eine Seite aufmachen or erfrischen, dann Chrome kommt mit (CryptSvc Anwendung erlischt).

Oder?

The silent assumption about IE first, is that it's used to access a secure page.  I can imagine there are scenarios where IE is open but not using CryptSvc and that an additional page access is needed to engage CryptSvc.

When you tested the IE first experiment, was this in a managed (AD) or unmanaged (home) environment?

BTW: I prefer to discuss these less technical issues in the help forum, where there is a larger audience, here
https://productforums.google.com/forum/#!topic/chrome/s5S1uPI0kMc

Old safe banking maybank2.com now

@sfor, other AD admins: If you get a chance, I need a second look at Lee's (aka le..@quarrybaykids.com) cryptsvc logon change with other domain user mods.  See #171 or here: 
https://productforums.google.com/d/msg/chrome/s5S1uPI0kMc/mib23iqtCAAJ

Although you don't use mandatory profiles, would Lee's changes have worked for you?  Or did you play thru similar scenarios and still have other side effects you couldn't fix?  

Do you see any security downside (over extension) to the domain user access extension or the CryptSvc desktop access?

Lee QBS reports that even with the other domain changes, changing the CryptSvc logon blocks updates from the WSUS server and device driver install by local admin - as reported here earlier.
11/22: https://productforums.google.com/d/msg/chrome/s5S1uPI0kMc/DnutgdQ3BwAJ

Looks like the CryptSvc logon change is not an option.

Correct - Local Services does fix the Chrome issue, but the side effects are unworkable.

any news about the result of analysing the process monitor files?

robert

Sorry for the delay, and thanks for the logs included in #169, #170, and #174. These are large logs, but they were consistent in their observable characteristics.

The behavior observed in the logs is:

In the Chrome browser process:

  1. Chrome invokes net::DoVerifyOnWorkerThread() [1], which results in invoking CertGetCertificateChain() from crypt32.dll.

  2. ... which invokes CCertChainEngine::GetChainContext from crypt32.dll

  3. ... which calls CCertChainEngine::Resync() from crypt32.dll

  4. ... which indirectly calls ResyncFromRegistry() from crypt32.dll

     Notably, this call happens roughly three times per CCertChainEngine::Resync() call in the logs that were analyzed.

  5. ... which calls IPR_DeleteUnprotectedRootsFromStore from crypt32.dll

     There is likely a bug here as to why this step happens multiple times per verification instead of just once or none at all.

At this point, IPR_DeleteUnprotectedRootsFromStore makes a synchronous RPC to CryptSvc and invokes s_SSCertProtectFunction() in cryptsvc.dll running under the CryptSvc process.

In the CryptSvc service process:

  6. s_SSCertProtectFunction() calls SrvGetProtectedRootInfo from crypt32.dll.

  7 ... which invokes SrvCreateProtectedRootSubKey() from crypt32.dll.

  8 ... which deletes, recreates, and resets permissions on the user's ProtectedRoots registry key [2]. The exact path of this key depends on whether it's a domain user or a local user. In logs #169 and #170 all operations involved in the deletion and recreation succeed.

These steps repeat a large number of times. E.g. The log in #170 indicates that DoVerifyOnWorkThread() was invoked 2226 times which resulted in the ProtectRoots key being deleted and re-created 5800 times, all within the span of 40 seconds. The log in #169 invokes DoVerifyOnWorkerThread() 8756 times also within a 40 second interval. This seems excessive.

It's notable that the DoVerifyOnWorkerThread() tasks were invoked on different threads on the worker pool in the Chrome browser process. However none of the calls overlap each other. This is indicative of the tasks being serialized by circumstance, not implementation. I.e. it's likely that all the tasks were the result of a single higher level retry loop. The invoker's stacks are not present in the logs which only sample each time some set of system calls are invoked. Neither are the return values of CertGetCertificateChain() invocation which is at the interface between Chrome and crypt32.

Based on that, I'd conclude that:

  * There doesn't seem to be a unconstrained loop in CryptSvc since each RPC completes within a reasonable amount of time. The constant deletion and recreation of ProtectedRoots is likely a bug though. So is the invocation of IPR_DeleteUnprotectedRootsFromStore multiple times during the construction of a single cert chain.

  * The massive number of net::DoVerifyOnWorkerThread tasks is very likely a bug. Each individual task completes in a reasonable amount of time modulo the synchronous wait while CryptSvc does its thing. This synchronous call and the implicit serialization of the net::DoVerifyOnWorkerThread tasks explains why the Chrome browser process doesn't get pegged the way CryptSvc does.

I didn't look too deeply into what might cause the certificate validation logic to be invoked in this manner. Our SSL folks can do a much more efficient job of figuring that out, so I'll yield to them :-)

Possible next steps:

* Since the return values of the CertGetCertificateChain and internal logic of Chrome wasn't captured in the ProcMon logs, it might be worthwhile getting a network log from Chrome from an affected system. Instructions for doing so can be found here: https://www.chromium.org/for-testers/providing-network-details

* Figure out why net::DoVerifyOnWorkerThread tasks are being posted this way.

[1]: https://cs.chromium.org/chromium/src/net/cert/multi_threaded_cert_verifier.cc?l=197&rcl=a4fc95699a752eb1f14d7e83f47e462cd6f242fc

[2]: https://blogs.technet.microsoft.com/leesteve/2017/05/24/demystification-of-the-protectedroots-registry-key/

asanka@ Thx, and on a Sunday nonetheless.  Thoughts on where to look for the problem..

The 3 logs are all for failure modes and from AD sites.
In the healthy case, either AD or local user, the looping does not occur, at least not noticeably.  

What we are trying to isolate is why some loop but not others.  For local (consumer) users, manually deleting HKCU...ProtectedRoots rebuilds and then Chrome will work.  The problem seems to be the ACE permissions on HKCU...ProtectedRoots.  Per asanka 8 ... above: the [ProtectedRoots] deletion and recreation succeed [but continue to repeat/loop]. 

The 3 logs represent 3 different Windows domain profile use cases:
11/20 #169 sfor@ AD profile: local
11/20 #170 off@ AD  profile: roaming
11/20 #174 le@ AD   profile: mandatory login, chrome profile roaming 

Note, Ranolph 10/12 #143 uses both roaming and mandatory profiles: the Ranolph roaming profiles work OK but the mandatory profiles loop.

The single user powershell repair script logs the initial Root and Protected ACLs before repair.  I have a small collection of logs for broken users, if this would help.

The failures seem to impact all Chromium based browsers, not just Chrome. Deleting/rebuilding ProtectedRoots fixes Chrome and the other user browsers as well.

In general, but maybe with a few caveats, starting Edge/IE before Chrome allows Chrome to work.  Per asanka's analysis above, probably, Edge/IE refreshes the certificate store and then Chrome doesn't need to refresh or sees a healthy set and doesn't refresh.

All: Please provide network logs per asanka's request above. Instructions:
 https://www.chromium.org/for-testers/providing-network-details

Larry - volunteer with a year's worth of notes..

AD Admins: FYI: resources, tips

A dedicated admin help forum is here (top)
  https://productforums.google.com/forum/m/#!forum/chrome-admins
or Admin topics
  https://productforums.google.com/forum/m/#!categories/chrome-admins
This admin topic targets the CryptSvc problem.  
  https://productforums.google.com/forum/m/#!category-topic/chrome-admins/r2A-ishIzrA

Creating new topics or comments in Chrome-admins may have more weight than posts here.

The admin CryptSvc topic, in addition to the regedit fix, mentions disabling the proxy auto detect in 
  Chrome Settings> Advanced> System> Proxy: disable auto-detect

I scanned the *Izra CryptSvc admin topic, but didn't see anything we haven't mentioned here, but you might see something that fits your particular environment.

attached is the network log from Chrome...

robert

Deleted: chrome-net-export-log.json 1.6 MB

chrome-net-export-log.json 1.6 MB View Download

#186: Thanks for the log! That gives us the stalling certificate verifier jobs and the certificates that it choked on. Interestingly it seems some certificates were successfully verified.

Hello, when can we expect a fix for this issue?

robert

I've spend a good number of hours tackling this issue & trying to find a workaround for that works for us.
We are using mandatory profiles & none the solutions above were working.

As far as i can tell, cryptsvc kept undoing all the fixes i did to the profile & i kept getting 'access denied' errors when trying to open the protectroots key in the hive.

eventually i restricted the permissions of cryptsvc, so it couldn't destory the fixes i've found here. this proved to be a working "solution for us".

attached you can find a pdf with a rough description how we managed to get it sorted

Deleted: Mandatory profile workaround for cryptsvc issue with chrome on w10 1803.pdf 573 KB

Mandatory profile workaround for cryptsvc issue with chrome on w10 1803.pdf 573 KB Download

We are using Roaming Profiles so no workaround for us now. I have sent all the necessary logs for analysis so it would be fine to get a feedback what's next or when we can expect a fix...

@robert
it's possible the changes i've come up with, work for roaming profiles aswell.
you'd need to find a way to script it, so you don't have to go over them 1-by-1

@hep
i have investigated some time for a solution but nothing works and in my opinion the only solution is a bug fix in chrome...
I have done all which the guys from google asked but since a few days ther is no feedback about the problem...
it is not my business to search for a workaround for a bug in a software - google should fix it - thats it!

hep#189 - Thanks, I appreciated the step-by-step digest.

I'm missing some pieces about which steps are necessary.  
Can you provide screenshots of the initial 
 ProtectedRoots(PR) permissions> Advanced for
   manprofa (bad mandatory) 
   manprofb (new user, works OK)

I think there is a way to stop CryptSvc and block restart while doing surgery
  services.msc> CryptSvc> Props> General> Startup type: disabled or manual
Would this have helped?
Once healthy, can you revert the CryptSvc PR permissions to the default full?

Lee (le#174) was eventually able get my script to work with mandatory profiles, look here, search for Lee QBS:
https://productforums.google.com/d/msg/chrome/s5S1uPI0kMc/bhIHtrsyDQAJ
Did you try this approach?

I am considering reworking the script(s) to have a username argument instead of requiring the target user to own the scripts folder.  
Would  that help you in the AD mandatory profiles environment?

@larrylac

-originally the "bad" mandatory did not have a ProtectRoots key (original_manprof_OFFLINE.jpg) in the registry. When the user logs in, it gets created automatically with bad permissions. 

- then i've tried to remove the entire 'systemcertificates' key from the bad manprof. When the user logs in, it gets created automatically with bad permissions.

summerized steps i took to get it working:
- i've imported a working 'SystemCertificates' key into the "bad" mandatory
- i've manually copied all the permissions & owners from a working profile to the mandatory hive.

at that point i hoped i'd have fixed it. unfortunately perms were still getting mangled once the user logged in. (no application was started before checking the permissions with regedit). 
I made a dozen attempts to find a way to get it working without choking cryptsvc.

- i've tried renaming ntuser.man --> ntuser.dat, logging off, renaming it back to ntuser.man. this gives the same result & a broken mandatory profile because the generic permissions are now modified towards the specific user.


- in the end i gave up and strangled cryptsvc so it couldn't destroy the permissions.

- to repeat myself: 'protectedroots' permissions get screwed up BEFORE ANY application is started manually. they get screwed upon login 


 

Deleted: original_manprof_OFFLINE.JPG 22.9 KB

	original_manprof_OFFLINE.JPG 22.9 KB View Download

Deleted: original_manprof_OFFLINE_root.JPG 110 KB

	original_manprof_OFFLINE_root.JPG 110 KB View Download

This issue is too big and difficult to handle in our roaming an mandatory AD environment. We are starting removing Google Chrome and replace it by Firefox.

I really hope Google will make a fix on a short notice. Otherwise we are stopping presenting Google Chrome to the school computers in our district. 

hep#196: Thx for the perms screenshot but the critical one is for the bad ProtectedRoots(PR) - unless it's the same as for Root.  When I look at my healthy PC the PR perms are different.

The #196 summary is what I understood from the .pdf.  Note, when you import the good SysCert thru PR from the new user hive, this only gets the data, the existing perms are not changed.  Since you didn't del the bad PR, its bad perms persist.

I didn't see an explicit mention of deleting PR (and then Root) and waiting for an auto-rebuild.  This has been the key repair step I've seen. 

Can you confirm that PR delete auto-rebuild failed?  It seems to work for le@ who has a similar mandatory AD environment.

FWIW: I think regedit needs CryptSvc authentication.  Login then open regedit is enough to trigger a refresh on PR.  Whether it's the login or another action which clobbers the PR perms is irrelevant, bad is bad.  But the regedit interaction makes it hard to track the perm damage creation.

I don't think this is an issue since you've really picked through the permissions so well, but if you login as admin, then look at HKCU, you could end up looking at the admin's HKCU, not the user's.  I find it's always safer to navigate thru HKU to the target user SID.  I sometimes create a debug string item under HKCU Certificates or Root for the target user, then from HKU verify I have the right SID key.

TIA

hep#194: Sorry, I missed the leading item
  -originally the "bad" mandatory did not have a ProtectRoots key
but I still can't confirm this from the screenshot 
since the Root key is not expanded.  

In the blank original mandatory profile, I would expect PR to be present but empty: with (only) Certificates Reg_Binary length 24

(and my previous #196 typo: should have also been hep#194:)

wdfshare@ #198, Did you try the registry repair, either manual or scripted? This should be simpler than re-image.  Microsoft updated to 1805 recently too.

For the Registry repair, look here
https://productforums.google.com/forum/#!msg/chrome/s5S1uPI0kMc/b2Mud3SKCgAJ

If you have questions on the registry repair, post to the help forum and I'll followup there.

This issue has only recently developed for me which I don't understand at all.  I installed an SSD drive and did a fresh install of W10 last year and everything's been like glass until recently.  In the last week whenever I open Chrome the CPU runs up to 100% with minimum being above 45% and memory gets devoured as well.  Stopping the Cryptsrv or whatever the hell that was worked for about one minute.  You wanna know my solution to this since Google doesn't want to fix their browser?  I uninstalled Chrome and have been using Edge.  The Google engineers are being pr*cks on this and Google deserves losing every user affected.  This has been a known issue since at least May of last year and Google just ignores us because "f*** Windows users"?  You seriously are going to force us to make register changes, use calendar to run scripts, and all the other crap you've got strung out above to address this rather than fixing it on your end?!?  Here's what that looks like to me:
"Hey, the dozen of us in this lab could fix this or we could just not and tell the tens of thousands of people affected to f*** off."
"Well, it's a lot easier to tell them to f*** off."
"OK, f*** off it is."
So, until you fix this, f*** Google.

Hello, when can we expect a fix for this issue?

robert

Dear

davidben@chromium.org
sindhu.chelamcherla@chromium.org 
robertshield@chromium.org 
kkaluri@chromium.org 
asanka@chromium.org 
rsleevi@chromium.org
georgesak@chromium.org
bheenan@chromium.org
susan.boorgula@chromium.org

Please tell us what we can expect in the matter of solving this issue?

Al least could some one from chromium DEV team give us information if this BUG is their problem or Microsoft?

The Windows 1809 Jan 2019 update KB4480116 fixed a similar Opera problem.
Any help for Chrome? Check Windows Settings> Update History to see if you have the KB4480116 update.

Opera fix reported here:
https://forums.opera.com/topic/30250/solved-back-to-firefox-due-to-opera-s-sluggishness/12

Thanks for everyone who has contributed.

Based on the investigation, we're going ahead and marking this WontFix. Based on our investigation, our view is that there's a corruption of registry permissions that causes core Windows services to enter into a busy loop. I'm not sure I've reached the same conclusions as from Comment #183; it appears that this loop is within the overall set of Crypt32.dll functionality (such as the retry), and not a Chrome-side loop.

I appreciate the work larryl... has done on this bug and assisting users in repairing the registry permissions. I've gone through the Logs in #169 and #170 again to make sure nothing is being overlooked here.

There are questions I don't have answers to:
1) Why is the users' registry getting corrupted in the first place (this seems strongly tied to a Microsoft software release, as noted in the subject)?
2) Why is CryptSvc unable to recover its settings? This is part of the core functionality of CryptSvc and IPR_DeleteUnprotectedRootsFromStore - it's meant to be able to recover from corruption of the authroot download, up to and including resync'ing from the copy of crypt32.dll.


What I see happen, from looking at the traces in #169 / #170:
- The initial call to CertGetCertificateChain is done on a general worker thread
- At some point, crypt32.dll decides during the ResyncFromRegistry call that it needs to invoke IPR_DeleteUnprotectedRootsFromStore (to force CryptSvc to re-create the registry store of trusted roots)
- CryptSvc is invoked, recreates this
- Verification continues
- Another worker thread starts verifying certificates (as we verify certificates in parallel on worker threads) while Worker 1 is still verifying
- Worker 2 decides it *also* needs to force a resync, which begins doing the same process
- Worker 1 detects that the registry key has changed. Crypt32.dll internally monitors the registry keys it uses for trust to pickup changes, so this triggers as CCertChainEngine::Resync() call, which internally calls ResyncFromRegistry(), thus causing another call to IPR_DeleteUnprotectedRootsFromStore
- Worker 1 now makes Worker 2 unhappy - and they keep ping-ponging between eachother, resync'ing and causing the other worker to do work
- This problem amplifies when more connections are made (thus causing more certificates to be verified) thus creating this storm.

The root of this storm is that Worker 1 and Worker 2 are not supposed to be resync'ing the registry hive that often. They "should" only do this if the registry hive is corrupted (although there's a variety of 'undocumented' flags to control this).

So either there is corruption which they can't recover from OR there's a set of flags being configured that are affecting things. Unfortunately, the registry bit being 'undocumented', it's difficult to find how best to confirm/rule-out; "generally" these sorts of flags will be reported in HKLM\SOFTWARE\Microsoft\Cryptography\OID\EncodingType 0\CertDllCreateCertificateChainEngine\Config as various configuration options off that tree, as well as HKLM\SOFTWARE\Microsoft\SystemCertificates\AuthRoot (and sub-trees). The question as to "Why might Edge work and Chromium-based browsers not" could be answered by Edge using an explicit configuration for the Chain Engine being created - or per-process configuration options that exist.

Enabling a CAPI2 Log ( https://blogs.msdn.microsoft.com/benjaminperkins/2013/09/30/enable-capi2-event-logging-to-troubleshoot-pki-and-ssl-certificate-issues/ ) could possibly figure out what CAPI calls are being invoked between Edge vs Chromium to determine if there is difference; however, the logging may not be granular enough for investigating this particular bit (since it's tied to chain engine internals). 

I realize this answer is probably particularly unsatisfying for folks who have been following along, hoping that some Chrome update will fix this. Unfortunately, we're treating this as an OS bug that cannot be reliably reproduced, and, due to the nearly 300 ways in which it can be customized/configured, is best dealt with by 'upstream' (in this case, Microsoft). 

@larrylac #204
Unfortunately KB4480116 update does not solve this, I'm on Windows 10 1809.
Also I've noticed that due to connection problems Chrome is also causing CPU load at 50% or more and it makes Windows very slow with some dramatically lowered responsiveness. It's caused by 'software_reporter_tool.exe' which I suppose is trying to connect with some google servers and it gets stuck also because of cryptsvc common problem...

re: Comment #206: Thanks for sharing that detail. It's interesting to note that the software_reporter_tool.exe uses a completely different networking stack than Chrome (it uses the Windows built-in WinHttp stack, source at https://cs.chromium.org/chromium/src/chrome/chrome_cleaner/http/http_agent_impl.cc?l=1&rcl=64b76696cded1502e1f9dd055dd3e2a51fa70f80 ).

This further supports that it is a Microsoft Windows' issue / issue with the underlying machine being in a 'bad state'. Event traces such as from comment #169/#170 could confirm this.

+people interested in potential software_reporter_tool hangs.

just for people reading this...

larry posted the current working fix is here.  Worked for ME!

https://productforums.google.com/forum/#!msg/chrome/s5S1uPI0kMc/PVBgVbx6DAAJ

Re: WontFix - There is a workaround for single end users.  It's not trivial, requires finding the solution and then wading in, which is probably <5% of single, consumer users.  Not good, but the ROI issues are clear.

The situation for managed, AD, enterprise or non-profit organizational environments, has no viable repair, particularly for mandatory profiles.  While the problem only affects some, and probably a small number of, managed systems, there is no option other than freezing Windows (LTSB) at 1709 or switching to another browser.  Windows 1809 may even exacerbate the risk: it may reintroduce the problem where it was already repaired in 1803.

Since the Chromium team can't repro this in house and debug it live, can this be done remotely at an AD site?  The AD admins would have the skills and tools needed to collaborate, and these are well managed, relatively clean environments.

I appreciate the difficulty/impossibility of tracking this through the 100's of undocumented Windows options and agree that part of this needs to be addressed through/with Microsoft. However I see no links or refs to Microsoft resources where AD admins can follow up.

For the AD environments, is Google really prepared to say 'If you have a cryptSvc storm, switch to another browser?'  That's the only option I see here.  This may be a small set of organizations, but it is only going to get larger with time.

rsleevi@ - Pls fwd to the Enterprise team. Any links to other Chrome AD admin support resources also welcome.