Failed to launch Chrome on Server 2016 DC (The specified module could not be found) |
||||||||||||
Issue descriptionChrome version: 67/68/69 OS version: Windows Server 2016 Datacenter/Core 10.0.14393 Case#: 16517177 Description: Chrome 68.0.3440.75 fails to install/launch on Server 2016 Data Center running Windows 10.0.14393. DETAILS: - Affecting all customer's Windows Server 2016 DC. - Other programs on the servers work fine, including Visual Studio Code which is built on Electron using the Chromium engine. - Other users have reported this issue: https://stackoverflow.com/questions/51860091/does-chromium-headless-work-on-windows-server-core-2016?rq=1 - Chrome 68 debug.log receives entry at each attempt to launch Chrome: [0725/201321.828:ERROR:main_dll_loader_win.cc(134)] Failed to load Chrome DLL from C:\Program Files (x86)\Google\Chrome\Application\68.0.3440.75\chrome.dll: The specified module could not be found. (0x7E) Steps to reproduce: - Install Chrome - Try to start Current Behavior / Reproduction: - Chrome doesn't start throwing error Expected Behavior: - Chrome starts Drive link to logs: N/A TROUBLESHOOTING STEPS TAKEN: 1. Uninstall Chrome. 2. Download the installation file directly from (also on a brand new server) https://enterprise.google.com/chrome/chrome-browser/ - Try standalone installer and offline installers as well. 3. Reboot computer. 4. Review antivirus, firewall, or parental control settings. 5. Try Microsoft tool to fix corrupted registry keys that block from installing programs: https://support.microsoft.com/en-us/help/17588/fix-problems-that-block-programs-from-being-installed-or-removed 6. Followed workaround found at crbug/786778 as it fixed the issue for most users running Canary 64: - Added the openvr_api.dll and openvr_api.pdb in the Chrome Application folder. DLLs are found at https://github.com/ValveSoftware/openvr/tree/master/bin. - Deleted ~ AppData\Local\Google\Chrome folder and reinstalled. 7. Tested on a new Windows Server 2016 Core instance stood up on Microsoft Azure (to eliminate any issues with the Amazon virtual machine image) and got the same result. 8. Updated to Chrome 69.0.3497.81. 9. Deleted Screenshots: https://drive.google.com/open?id=1g_sQx2Q0PjZId4khqYm97EtB2X2_XX5e https://drive.google.com/open?id=16YC_qN4MD5FRmstsxvUOqzXg99RzlEIj (Dependency Walker) Stack strace: https://drive.google.com/open?id=1_dsZwhGb1LvUh6pNh1RMW6thhb6WU3ZY WinDbg: https://drive.google.com/open?id=1wFlZD0BREvOki9NrTwY_CnIlcJSVPB6I PML files: https://drive.google.com/open?id=14nP1t5MRVGj0uY7eL4qWIauHR77pZcjL
,
Sep 30
,
Oct 2
Customer said this was working in version 67, so the repro steps are: 1. Open Google Chrome 67.0.3396.99 2. Run update via chrome://settings/help 3. Updates to version Google Chrome 68.0.3440.75 4. Click relaunch. 5. Chrome does not launch. OTHER SIMILAR CRBUGS: crbug.com/786778 crbug.com/757336 crbug.com/757483
,
Oct 2
privard@ can we determine if Windows 2016 Core is actually supported?
,
Oct 2
Hi Team, we received reports from another Enterprise customer getting the same error message for version 69 on windows Server 2016 Core: Case #: 17087413 Error in debug.log: [1002/102047.295:ERROR:main_dll_loader_win.cc(134)] Failed to load Chrome DLL from C:\Program Files (x86)\Google\Chrome\Application\69.0.3497.100\chrome.dll: The specified module could not be found. (0x7E)
,
Oct 2
interesting. can they run an sxstrace? sxstrace trace /logfile:sxstrace.etl [attempt to start chrome] sxstrace stoptrace (if needed) Then attach the trace file to this bug?
,
Oct 5
sxtrace log has been provided: https://drive.google.com/open?id=1LI1QzIvFh05vk1BzxiUkt6QgMTAMS-JW
,
Oct 5
,
Oct 8
can we get an update on this?
,
Oct 8
Hi sorry I thought I replied last week (I must have had a draft reply that got eaten) This sxstrace does not contain the trace for chrome.exe but for cscript.exe, so isn't helpful, unfortunately. Let's try using dependency walker instead: 1. Download depends.exe from http://www.dependencywalker.com/ 2. Open chrome.exe in dependency walker (and wait a long time, it takes a very long time to parse) 3. File -> Save As -> change type to txt and save as chrome.txt Attach chrome.txt here.
,
Oct 8
Please confirm - does 69.0.3497.81 work but 68.0.3440.75 does not?
,
Oct 8
DepWalker screenshot is already attached in the beginning. I'll re-request sxstrace log from customer
,
Oct 8
yes I saw the depwalker screenshot but that does not contain the full dump just the first page. Please follow steps in #10
,
Oct 8
I'm installing WS 2016 Datacenter and will see if I can repro this and work out what's going on.
,
Oct 8
I can reproduce this. I am debugging.
,
Oct 8
This is likely because of https://chromium-review.googlesource.com/c/chromium/src/+/1065918 added two calls to msdmo.dll!MoFreeMediaType msdmo.dll!MoInitMediaType msdmo.dll is not available on this OS.
,
Oct 8
Ah, I should have caught that new dependency in the CL :( Henrik - can we wire this up in a way that dynamically loads the library + getprocaddress, or can the dependency (and possibly the CL) be reverted?
,
Oct 8
can confirm that adding a delay load dep fixes the startup issues. https://chromium-review.googlesource.com/c/chromium/src/+/1269178 chrome doesn't seem to actually load on my Windows Server Core install, but I'm not sure it's related to msdmo, I think it's something else (unrelated to the delay load).
,
Oct 8
my crash is unrelated - it's issue 893253 . I think crrev.com/1269178 should be safe to land to fix this issue causing startup to fail, although obviously "audio voice capture DMO" will not work on this platform due to lack of support, so the code should probably handle that gracefully. Assigning to grunell@ to handle. Free free to steal my CL.
,
Oct 8
For Chrome to start, the delay load dep also has to be fixed in chrome_child.dll (I've updated my CL). This is worrying because chrome_child is running sandboxed and should not be pulling in dependencies that won't function inside the sandbox... I did check in IDA and both functions are unreferenced in the chrome_child.dll - so the CL really should just remove these dependencies. perhaps audio code could be split into parts that are only called from browser vs renderer? THe path of this dep appears to be: //chrome:chrome_child --[private]--> //chrome:child_dependencies --[public]--> //chrome/common:common --[public]--> //media:media --[public]--> //media/audio:audio I'm going to go ahead and add a test or a presubmit to try and detect these type of issues in the future.
,
Oct 8
oh wait IDA hadn't finished parsing chrome_child.dll previously. In fact, there are XREFs to both of these imports via callpath: media::WASAPIAudioInputStream::Open media::WASAPIAudioInputStream::InitializeDmo media::WASAPIAudioInputStream::SetDmoFormat I can't find a call to open() inside chrome_child.dll though, looks like it's going via a vcall from https://cs.chromium.org/chromium/src/services/audio/output_controller.cc?l=200 which is why the linker did not discard this import.
,
Oct 9
So, is the CL in #18 good to go apart from handling the lack of support in a suitable way?
,
Oct 9
#18 was enough to verify that these msdmo.dll calls were causing the startup issue on Server 2018, but the delay load implementation will be raising an exception if it can't load the DLL, which is a Bad Thing. The correct solution here is to LoadLibrary the msdmo.dll and fail gracefully if the two procs can't be resolved. I have tried running under a debugger on my Server 2018 machine to see if I could get the exception, but a breakpoint set on chrome!media::WASAPIAudioInputStream::InitializeDmo is never hit, so I think perhaps the feature isn't enabled, or something?
,
Oct 10
IIUC there was consensus that the DMO would not be used. Should we instead remove the dep?
,
Oct 10
The "DMO echo canceller" has to be enabled via a getUserMedia constraint that is available under an origin trial. You'd normally not hit those functions. Yes, the consensus was that it underperforms and should be removed eventually. (There were some reasons for keeping it for now, but not particularly strong ones.) Given this issue, I think it should just be removed right away. Even if loading the dll and handling failure of that would be simple I don't think it's worth it. There also the risk of further problems not yet known or bugs in the code that needs to added. Does anyone oppose removing it?
,
Oct 10
I don't oppose removing it, but I also don't think it's high priority compared to other things we want to do. If it's easy and we don't need the fix immediately (as in: for merging somewhere) I agree we should remove it rather than work around it not being available.
,
Oct 10
+1 for removing
,
Oct 10
It should not be severe enough to warrant a merge to M70, which has past stable cut. We agreed that Oskar will remove the code - I have other things for M71.
,
Oct 10
Can someone on the enterprise team comment on the priority of this bug and whether customers would be happy to wait until M71 for a Chrome that will launch on Windows Server 2018 Datacenter/Core.
,
Oct 11
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b67648ad60a925974ba5ec5437fe946ea269ab2a commit b67648ad60a925974ba5ec5437fe946ea269ab2a Author: Oskar Sundbom <ossu@chromium.org> Date: Thu Oct 11 11:13:19 2018 Win: Remove support for the voice processing DMO echo canceller After experimenting with it, we've decided to not move forward. As the implementation is causing some issues, it's best to just remove it. Bug: 888783 , 845187 Cq-Include-Trybots: luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel Change-Id: I4ef9622340266930737f9a2597c33375702f53b8 Reviewed-on: https://chromium-review.googlesource.com/c/1273069 Commit-Queue: Oskar Sundbom <ossu@chromium.org> Reviewed-by: Henrik Andreasson <henrika@chromium.org> Reviewed-by: Henrik Grunell <grunell@chromium.org> Cr-Commit-Position: refs/heads/master@{#598722} [modify] https://crrev.com/b67648ad60a925974ba5ec5437fe946ea269ab2a/media/audio/BUILD.gn [modify] https://crrev.com/b67648ad60a925974ba5ec5437fe946ea269ab2a/media/audio/win/audio_low_latency_input_win.cc [modify] https://crrev.com/b67648ad60a925974ba5ec5437fe946ea269ab2a/media/audio/win/audio_low_latency_input_win.h [modify] https://crrev.com/b67648ad60a925974ba5ec5437fe946ea269ab2a/media/audio/win/audio_low_latency_input_win_unittest.cc [modify] https://crrev.com/b67648ad60a925974ba5ec5437fe946ea269ab2a/media/audio/win/audio_manager_win.cc [modify] https://crrev.com/b67648ad60a925974ba5ec5437fe946ea269ab2a/media/audio/win/core_audio_util_win.cc [modify] https://crrev.com/b67648ad60a925974ba5ec5437fe946ea269ab2a/media/audio/win/core_audio_util_win.h
,
Jan 11
Hello! This bug is receiving this notice because there has been no acknowledgment of its existence in quite a bit of time. - If you are currently working on this bug, please provide an update. - If you are currently affected by this bug, please update with your current symptoms and relevant logs. If there has been no updates provided by EOD Thursday, 01/17/19 (5pm EST), this bug will be archived and can be re-opened at any time deemed necessary. Thank you!
,
Jan 14
I expect this issue was fixed by the change in #30, could someone with access to the appropriate Windows version (wfh@?) take a look?
,
Jan 14
Verified that Chrome stable 71.0.3578.98 works fine on Windows 2016 datacenter 10.0.14393 |
||||||||||||
►
Sign in to add a comment |
||||||||||||
Comment 1 by ykrychala@chromium.org
, Sep 24Labels: OS-Windows