New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 888783 link

Starred by 3 users

Issue metadata

Status: Verified
Owner:
Closed: Jan 14
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 2
Type: Bug


Show other hotlists

Hotlists containing this issue:
Chrome-Bug-Cleanup


Sign in to add a comment

Failed to launch Chrome on Server 2016 DC (The specified module could not be found)

Project Member Reported by ykrychala@google.com, Sep 24

Issue description

Chrome version: 67/68/69
OS version: Windows Server 2016 Datacenter/Core 10.0.14393 
Case#: 16517177

Description:
Chrome 68.0.3440.75 fails to install/launch on Server 2016 Data Center running Windows 10.0.14393. 
DETAILS: 
- Affecting all customer's Windows Server 2016 DC. 
- Other programs on the servers work fine, including Visual Studio Code which is built on Electron using the Chromium engine. 
- Other users have reported this issue: https://stackoverflow.com/questions/51860091/does-chromium-headless-work-on-windows-server-core-2016?rq=1 

- Chrome 68 debug.log receives entry at each attempt to launch Chrome: 
[0725/201321.828:ERROR:main_dll_loader_win.cc(134)] Failed to load Chrome DLL from C:\Program Files (x86)\Google\Chrome\Application\68.0.3440.75\chrome.dll: The specified module could not be found. (0x7E) 


Steps to reproduce: 
- Install Chrome
- Try to start

Current Behavior / Reproduction: 
- Chrome doesn't start throwing error

Expected Behavior: 
- Chrome starts

Drive link to logs: 
N/A


TROUBLESHOOTING STEPS TAKEN: 
1. Uninstall Chrome. 
2. Download the installation file directly from (also on a brand new server) https://enterprise.google.com/chrome/chrome-browser/ 
- Try standalone installer and offline installers as well. 
3. Reboot computer. 
4. Review antivirus, firewall, or parental control settings. 
5. Try Microsoft tool to fix corrupted registry keys that block from installing programs: https://support.microsoft.com/en-us/help/17588/fix-problems-that-block-programs-from-being-installed-or-removed 
6. Followed workaround found at crbug/786778 as it fixed the issue for most users running Canary 64: 
- Added the openvr_api.dll and openvr_api.pdb in the Chrome Application folder. DLLs are found at https://github.com/ValveSoftware/openvr/tree/master/bin. 
- Deleted ~ AppData\Local\Google\Chrome folder and reinstalled. 
7. Tested on a new Windows Server 2016 Core instance stood up on Microsoft Azure (to eliminate any issues with the Amazon virtual machine image) and got the same result. 
8. Updated to Chrome 69.0.3497.81. 
9. Deleted 

Screenshots:
https://drive.google.com/open?id=1g_sQx2Q0PjZId4khqYm97EtB2X2_XX5e
https://drive.google.com/open?id=16YC_qN4MD5FRmstsxvUOqzXg99RzlEIj (Dependency Walker)

Stack strace:
https://drive.google.com/open?id=1_dsZwhGb1LvUh6pNh1RMW6thhb6WU3ZY

WinDbg:
https://drive.google.com/open?id=1wFlZD0BREvOki9NrTwY_CnIlcJSVPB6I

PML files:
https://drive.google.com/open?id=14nP1t5MRVGj0uY7eL4qWIauHR77pZcjL

 
Components: Internals>Core
Labels: OS-Windows
It looks like Chrome.dll requires some DLLs as a dependency which are missing in Windows 10 DC.
Do we support Chrome running on this type of Windows install?
Labels: Needs-Milestone
Customer said this was working in version 67, so the repro steps are:

1. Open Google Chrome 67.0.3396.99 
2. Run update via chrome://settings/help 
3. Updates to version Google Chrome 68.0.3440.75 
4. Click relaunch. 
5. Chrome does not launch. 

OTHER SIMILAR CRBUGS: 
 crbug.com/786778  
 crbug.com/757336  
 crbug.com/757483  
Owner: privard@chromium.org
privard@ can we determine if Windows 2016 Core is actually supported?
Hi Team, we received reports from another Enterprise customer getting the same error message for version 69 on windows Server 2016 Core:

Case #: 17087413

Error in debug.log: 
[1002/102047.295:ERROR:main_dll_loader_win.cc(134)] Failed to load Chrome DLL from C:\Program Files (x86)\Google\Chrome\Application\69.0.3497.100\chrome.dll: The specified module could not be found. (0x7E)
interesting. can they run an sxstrace?

sxstrace trace /logfile:sxstrace.etl  

[attempt to start chrome]

sxstrace stoptrace (if needed)

Then attach the trace file to this bug?

Cc: wfh@chromium.org
Owner: wfh@chromium.org
can we get an update on this?
Hi sorry I thought I replied last week (I must have had a draft reply that got eaten)

This sxstrace does not contain the trace for chrome.exe but for cscript.exe, so isn't helpful, unfortunately.

Let's try using dependency walker instead:

1. Download depends.exe from http://www.dependencywalker.com/
2. Open chrome.exe in dependency walker (and wait a long time, it takes a very long time to parse)
3. File -> Save As -> change type to txt and save as chrome.txt

Attach chrome.txt here.
Please confirm - does 69.0.3497.81 work but 68.0.3440.75 does not?
DepWalker screenshot is already attached in the beginning.
I'll re-request sxstrace log from customer
yes I saw the depwalker screenshot but that does not contain the full dump just the first page. Please follow steps in #10
Status: Started (was: Unconfirmed)
I'm installing WS 2016 Datacenter and will see if I can repro this and work out what's going on.
I can reproduce this. I am debugging.
Cc: grunell@chromium.org tommi@chromium.org
This is likely because of

https://chromium-review.googlesource.com/c/chromium/src/+/1065918

added two calls to

msdmo.dll!MoFreeMediaType
msdmo.dll!MoInitMediaType

msdmo.dll is not available on this OS.



Ah, I should have caught that new dependency in the CL :(

Henrik - can we wire this up in a way that dynamically loads the library + getprocaddress, or can the dependency (and possibly the CL) be reverted?
can confirm that adding a delay load dep fixes the startup issues.

https://chromium-review.googlesource.com/c/chromium/src/+/1269178

chrome doesn't seem to actually load on my Windows Server Core install, but I'm not sure it's related to msdmo, I think it's something else (unrelated to the delay load).
Cc: -grunell@chromium.org ossu@chromium.org solenberg@chromium.org
Components: Blink>WebRTC>Audio Internals>Media>Audio
Owner: grunell@chromium.org
Status: Assigned (was: Started)
my crash is unrelated - it's  issue 893253 . I think crrev.com/1269178 should be safe to land to fix this issue causing startup to fail, although obviously "audio voice capture DMO" will not work on this platform due to lack of support, so the code should probably handle that gracefully.

Assigning to grunell@ to handle. Free free to steal my CL.
For Chrome to start, the delay load dep also has to be fixed in chrome_child.dll (I've updated my CL). This is worrying because chrome_child is running sandboxed and should not be pulling in dependencies that won't function inside the sandbox... I did check in IDA and both functions are unreferenced in the chrome_child.dll - so the CL really should just remove these dependencies.

perhaps audio code could be split into parts that are only called from browser vs renderer?

THe path of this dep appears to be:

//chrome:chrome_child --[private]-->
//chrome:child_dependencies --[public]-->
//chrome/common:common --[public]-->
//media:media --[public]-->
//media/audio:audio

I'm going to go ahead and add a test or a presubmit to try and detect these type of issues in the future.
oh wait IDA hadn't finished parsing chrome_child.dll previously. In fact, there are XREFs to both of these imports via callpath:

media::WASAPIAudioInputStream::Open
media::WASAPIAudioInputStream::InitializeDmo
media::WASAPIAudioInputStream::SetDmoFormat

I can't find a call to open() inside chrome_child.dll though, looks like it's going via a vcall from https://cs.chromium.org/chromium/src/services/audio/output_controller.cc?l=200 which is why the linker did not discard this import.


So, is the CL in #18 good to go apart from handling the lack of support in a suitable way?
#18 was enough to verify that these msdmo.dll calls were causing the startup issue on Server 2018, but the delay load implementation will be raising an exception if it can't load the DLL, which is a Bad Thing.

The correct solution here is to LoadLibrary the msdmo.dll and fail gracefully if the two procs can't be resolved.

I have tried running under a debugger on my Server 2018 machine to see if I could get the exception, but a breakpoint set on chrome!media::WASAPIAudioInputStream::InitializeDmo is never hit, so I think perhaps the feature isn't enabled, or something?
IIUC there was consensus that the DMO would not be used. Should we instead remove the dep? 
Cc: henrika@chromium.org
The "DMO echo canceller" has to be enabled via a getUserMedia constraint that is available under an origin trial. You'd normally not hit those functions.

Yes, the consensus was that it underperforms and should be removed eventually. (There were some reasons for keeping it for now, but not particularly strong ones.) Given this issue, I think it should just be removed right away. Even if loading the dll and handling failure of that would be simple I don't think it's worth it. There also the risk of further problems not yet known or bugs in the code that needs to added.

Does anyone oppose removing it?
I don't oppose removing it, but I also don't think it's high priority compared to other things we want to do. If it's easy and we don't need the fix immediately (as in: for merging somewhere) I agree we should remove it rather than work around it not being available.
+1 for removing 
Cc: grunell@chromium.org
Owner: ossu@chromium.org
It should not be severe enough to warrant a merge to M70, which has past stable cut. We agreed that Oskar will remove the code - I have other things for M71.
Can someone on the enterprise team comment on the priority of this bug and whether customers would be happy to wait until M71 for a Chrome that will launch on Windows Server 2018 Datacenter/Core.
Project Member

Comment 30 by bugdroid1@chromium.org, Oct 11

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b67648ad60a925974ba5ec5437fe946ea269ab2a

commit b67648ad60a925974ba5ec5437fe946ea269ab2a
Author: Oskar Sundbom <ossu@chromium.org>
Date: Thu Oct 11 11:13:19 2018

Win: Remove support for the voice processing DMO echo canceller

After experimenting with it, we've decided to not move forward. As the
implementation is causing some issues, it's best to just remove it.

Bug:  888783 ,  845187 
Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel
Change-Id: I4ef9622340266930737f9a2597c33375702f53b8
Reviewed-on: https://chromium-review.googlesource.com/c/1273069
Commit-Queue: Oskar Sundbom <ossu@chromium.org>
Reviewed-by: Henrik Andreasson <henrika@chromium.org>
Reviewed-by: Henrik Grunell <grunell@chromium.org>
Cr-Commit-Position: refs/heads/master@{#598722}
[modify] https://crrev.com/b67648ad60a925974ba5ec5437fe946ea269ab2a/media/audio/BUILD.gn
[modify] https://crrev.com/b67648ad60a925974ba5ec5437fe946ea269ab2a/media/audio/win/audio_low_latency_input_win.cc
[modify] https://crrev.com/b67648ad60a925974ba5ec5437fe946ea269ab2a/media/audio/win/audio_low_latency_input_win.h
[modify] https://crrev.com/b67648ad60a925974ba5ec5437fe946ea269ab2a/media/audio/win/audio_low_latency_input_win_unittest.cc
[modify] https://crrev.com/b67648ad60a925974ba5ec5437fe946ea269ab2a/media/audio/win/audio_manager_win.cc
[modify] https://crrev.com/b67648ad60a925974ba5ec5437fe946ea269ab2a/media/audio/win/core_audio_util_win.cc
[modify] https://crrev.com/b67648ad60a925974ba5ec5437fe946ea269ab2a/media/audio/win/core_audio_util_win.h

Hello!
This bug is receiving this notice because there has been no acknowledgment of its existence in quite a bit of time.
- If you are currently working on this bug, please provide an update.
- If you are currently affected by this bug, please update with your current symptoms and relevant logs.

If there has been no updates provided by EOD Thursday, 01/17/19 (5pm EST), this bug will be archived and can be re-opened at any time deemed necessary.

Thank you!
Status: Fixed (was: Assigned)
I expect this issue was fixed by the change in #30, could someone with access to the appropriate Windows version (wfh@?) take a look?
Status: Verified (was: Fixed)
Verified that Chrome stable 71.0.3578.98 works fine on Windows 2016 datacenter 10.0.14393

Sign in to add a comment