Attaching the last bits of the update log.

Deleted: Log.txt 85.3 KB

Log.txt 85.3 KB View Download

60 = SETUP_SINGLETON_ACQUISITION_FAILED. this means another setup.exe was doing something and didn't exit quickly.

could you provide %TEMP%\chrome_installer.log? maybe it says what it was doing.

assigning to fdoray@ to help diagnose.

chrome_installer.log attached.

Deleted: chrome_installer.log 7.9 KB

chrome_installer.log 7.9 KB View Download

Just ran into this again. Attached the logs.

Deleted: chrome_installer.log 14.8 KB

chrome_installer.log 14.8 KB View Download

Are any version directories still present in C:\Users\robliao\AppData\Local\Google\Chrome SxS\Application? What are the version #s in *chrome*.exe?

I don't have the repro anymore, but it was a similar story. chrome.exe was ahead two builds from the only version directory available.

This sounds like it could be caused by MoveFileEx hanging in the kernel as I mentioned on internal comms. If you notice it happen again, see if you have a stray setup.exe that can't be killed.

This is on Windows 10, yes? I have a report from an admin that also indicates a hang in FltMgr during install. This is not so good...

Yep, this is on Win10. I haven't seen it happen again recently, but I'll ping back and see what I can dig up if it does.

I've been commenting a thread in the Chrome Admin Forums, but thought I'd chime in here. I work at a University and we have had over 120 instances of this issue over the past 2 months.

I've captured a process dump of the hung setup.exe and you can see in my stack trace that it has indeed hung doing a MoveFileEx kernel call.
Today I've also captured a kernel dump on an affected machine incase it helps. Looking Memory.zip

https://drive.google.com/drive/folders/0ByWCZxunXZeNX2xVMHRvQTNZWDA?usp=sharing

Thanks. We'll try to get MS to investigate this.

Any update on this? It is still affecting some users here from time to time.

Just an FYI, I'm poised to open my own premier support case with MS on this. Any endorsement from those on this bug or do you not thing there is any value?

Not being able to acquire the SetupSingleton should prevent updates, but I don't understand why it would prevent Chrome from being launched.

Should the setup process have a hang watcher and be killed if it is unresponsive for some time?

Chrome can't launch because one of the MoveFileEx operations that moves one of the installation directories is getting wedged in the kernel. Somehow the version directory for the current chrome.exe isn't present, so chrome.dll can't be found. At least that's what I remember -- it's been a while since my machines have been hit by this, so I haven't been able to investigate deeper. The hung process cannot be killed since it's deep in the kernel -- only a restart will get rid of it.

Please let us know if you open an issue with MS, thanks.

Is the source available for the setup/GoogleUpdate program so I can attempt to replicate the issue?

The call that hangs in the kernel is moving a directory. It's essentially:

MoveFileExW(
    "C:\Users\USER\AppData\Local\Google\Chrome SxS\Temp\sourceXXXX_YYYY\Chrome-bin\W.X.Y.Z",
    "C:\Users\USER\AppData\Local\Google\Chrome SxS\Application\W.X.Y.Z",
    MOVEFILE_COPY_ALLOWED | MOVEFILE_REPLACE_EXISTING);

The move takes place when this work item in Chrome's installer is run: https://cs.chromium.org/chromium/src/chrome/installer/setup/install_worker.cc?sq=package:chromium&dr&l=259.

Hope this helps. We're also working on getting data to MS to help them diagnose the problem. Thanks.

What versions of Windows 10 has this been seen on? The kernel dump and process dump both appear to be from Anniversary Edition (14393) - has it been seen on any other versions?

I filed support request 117100416445054 with Microsoft.

I think my machine was already on Creators Update (1703) the last time I saw it, but I haven't had a repro in a little while. Anyone else?

Microsoft is investigating and it looks like a fltmgr callback to the wof.sys driver is not completing correctly for reasons not yet determined.

Some of the state has already disappeared by the time the kernel dump is recorded. Would it be possible to get a procmon trace of this issue happening?

In particular, it appears that once the issue has happened once it will continue happening until the machine is rebooted - is that correct? If so then it should be possible to trigger the hang on-demand and get a procmon trace.

This used to happen a lot more often but hasn't happened as of late for me :-/.

I'll see if I can collect some info when it does, but I don't know when that will be.

I have previously been able to get a procmon trace. It's not easy because you want to catch it the first time it's run. Unfortunately I think I deleted it as I could not find anything helpful in it. I will try and get one again.

Yes you are correct about the reboot. And then usually after the reboot, upgrade succeeds without issue, that's why the trace is hard to get.

Could you share your MS Case ID with me? I've got an open premier support ticket and my support engineer would like to collaborate with the other team.

The MS Case ID is 117100416445054

Well, next time anybody gets in this state if you could procmon a subsequent failure and be sure to save with "All events" that would be very helpful.

Ping! Procmon trace of the hang happening?

Another kernel dump would be helpful as well. Thanks.

Microsoft have asked me for one too. I have captured it before and will endevour to do so again.

"This is a code defect that was fixed in Redstone 3 / 1709 release.  This is also already in the pipeline to be backported to RS1, RS2, Server 2012 R2 and Server 2012 and those are all scheduled for the November Preview release (release 11/21)."

They also said that the deadlock is triggered by low-priority I/O.

It sounds like this defect has been around for a while. I'm asking for more details but until the fix ships it seems like the only mitigation would be to avoid low priority I/O. Low-priority I/O helps stop the Chrome update process from interfering with other tasks so avoiding low-priority I/O would potentially make user's machines less responsive during upgrades. Plus, there would be some risk associated with changing the updater temporarily while waiting for the fix.

The fix shipped in Windows 10 Creators Update and should ship at some point to RS1, RS2, and other variants. Has the issue been resolved? If not please let me know what versions of Windows it is being seen on.

I will try to get clarification of when the fix will ship for RS1 and RS2.

Clarification: The fix should have shipped for RS2 (Creators Update) on November 14th. The fix will ship for RS1 (Anniversary Edition), Windows Server 2012, Windows 8.1, and Windows Server 2012 R2 with the December updates.

So, most Windows 10 variants have been fixed for two weeks or longer, and more fixes will ship in a few weeks.

Closing this out since there's nothing more for us to do. Hopefully all impacted users have picked up MS's fixes.