New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 814641 link

Starred by 4 users

Issue metadata

Status: Fixed
Owner:
Closed: Feb 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug-Regression



Sign in to add a comment

Regression:[Win-7] All Chrome tabs are getting crashed after launching the 66.0.3352.0 (64-bit) build.

Reported by dchau...@etouch.net, Feb 22 2018

Issue description

Chrome Version: 66.0.3352.0 (Official Build) Revision db148559378834dcddbc7066d6f899b5e227d426-refs/heads/master@{#538313}
OS: Windows-7

What steps will reproduce the problem?
1. Freshly install and launch 66.0.3352.0 Chrome.
2. Observe.

All Chrome tabs are getting crashed.
Chrome should not crashed.

This is a regression issue, broken in M-66, will soon update the other info.
 

Comment 1 by dchau...@etouch.net, Feb 22 2018

Cc: pbomm...@chromium.org
Components: UI>Browser
Labels: RegressedIn-66 FoundIn-66 Target-66 Stability-Crash HasTestcase hasbisect
Owner: finnur@chromium.org
Status: Assigned (was: Unconfirmed)
Summary: Regression:[Win-7] All Chrome tabs are getting crashed after launching the 66.0.3352.0 (64-bit) build. (was: Regression: All Chrome tabs are getting crashed.)
Below is manual regression range for the above issue.

Good build: 66.0.3350.0 (Revision : 537343)
Bad build: 66.0.3352.0 (Revision : 538313)

Unable to provide the bisect using per-revision scripts as all Chrome builds are getting crashed from tool and unable to provide using normal bisect script as getting "runtime error". Hence providing ChangeLog URL:

ChangeLog URL:
https://chromium.googlesource.com/chromium/src/+log/66.0.3350.0..66.0.3352.0?pretty=fuller&n=10000

Suspecting: https://chromium.googlesource.com/chromium/src/+/aa525adf529ce884bdcb06dbd451444aaf294eb1

@finnur: Could you please look into the issue, pardon me if it has nothing to do with your changes and if possible please assign it to concern owner.

Note :
1. This issue is only reproducible on 66.0.3352.0 (Official Build) (64-bit) Chrome build only on Windows-7 machine and it is not reproducible on 66.0.3352.0 (Official Build) (32-bit) Chrome.
2. Unable to provide crash ids as all the tabs are getting crashed.
3. This issue is not reproducible on Windows(8.1,10), Mac(10.12.6, 10.13.1, 10.13.4) and Linux(14.04 LTS) machines.
4. This issue is also reproducible on Windows-8 machine.

Kindly review the attached screen-cast for reference.

Thank you.
Actual behavior.mp4
688 KB View Download
Expected behavior.mp4
698 KB View Download

Comment 2 by finnur@chromium.org, Feb 22 2018

Owner: ----
Status: Untriaged (was: Assigned)
This crash is on Win7 but the code I was changing only runs on Windows 10 and above, and it is also behind a Feature flag (which is off by default). Not sure who is appropriate to look at this. Marking for triage.
Cc: abdulsyed@chromium.org gov...@chromium.org
Labels: ReleaseBlock-Dev
Adding release blocker label for this issue.Please reduce priority or remove if not the case.

Thank You!
So far I wasn't able to reproduce checked on 4 different Windows 7 and 10 machines. 
Unable to reproduce the issue on Chrome 66.0.3352.0/CrOS 10429.0.0 - Kip
Cc: pabrai@chromium.org
dchaubey@ if possible can you please let us know below details :

1. Were you able to reproduce the issue on other Windows 7 machine.
2. If not can you please provide the details of the machine where you were able to reproduce this issue.

Comment 8 by wfh@chromium.org, Feb 22 2018

If you are able to repro please start fresh chrome with additional command line

--enable-logging=stderr --v=1 --vmodule=metrics=2 > log.txt 2>&1

Then exit chrome and attach log.txt
What happens if you run Chrome with --no-sandbox? We don't recommend this in general but as a quick test it could help us determine whether this crash is due to a recent sandbox related change.

With response to comment #7:
Re-tested this issue on three Win-7 machines using Chrome build # 66.0.3352.0 (64-bit) and able to reproduce the issue. 

NOTE: Issue is only reproducible on Chrome 66.0.3352.0 (64-bit)  and the same crash is not observed in Chrome 66.0.3352.0 (32-bit) build.

With response to comment #8:
Enabled the flag mentioned in C#8 (--enable-logging=stderr --v=1 --vmodule=metrics=2 > log.txt 2>&1) and got the following log.txt file

With response to comment #9:
Chrome doesn't crash and works properly when we run the Chrome with flag --no-sandbox
log.txt
320 KB View Download

Comment 11 by grt@chromium.org, Feb 23 2018

Cc: forshaw@chromium.org brucedaw...@chromium.org
Bruce and James: any chance that r538307 could be related?
I suspect that r538307 is the problem. That change is why I suggested --no-sandbox, and the fact that --no-sandbox avoids the crash increases the suspicion.

I'll revert that change and we can monitor.

It would be wonderful to see what happens when canary is launched under a debugger. If somebody who can reproduce the problem with "windbg -g -G -o chrome.exe" and report back on the error and other debug information that would be very helpful.

If somebody does run chrome.exe under windbg with the command-line listed in the previous comment and they catch the crash then it would be very helpful if a minidump of the first crashed process could be grabbed. This command will do the trick:

.dump c:\chrome.dmp

Adjust the path to save it wherever is most convenient, perhaps in your documents directory, then attach it to this bug. Thanks!

Comment 14 by wfh@chromium.org, Feb 23 2018

re:10 it looks like your GPU process is crashing a lot... Can you go to chrome://local-state and check if you have anything listed in "enabled_labs_experiments" field?

Comment 15 by wfh@chromium.org, Feb 23 2018

oh nvm if this is happening on win8 (#c1) then it can't be a lab experiment. I agree with bruce's diagnosis, but it would be useful to have a local repro - perhaps check in the crash dumps folder [1] to see if there are any .dmp files from the same time as the crash, and attach here or via google drive.

[1] try both:

%localappdata%\Google\Chrome\User Data\Crashpad\reports

%localappdata%\Google\Chrome SxS\User Data\Crashpad\reports


Comment 16 by wfh@chromium.org, Feb 23 2018

...or just load chrome with --no-sandbox and check for crashes in chrome://crashes - probably easier :)
I'm able to launch canary with --no-sandbox, but not without.

What are the security implications in using that until this is fixed?
I wouldn't recommend using Chrome without the sandbox. Can you try using a different channel until tomorrow's canary comes out? Stable, dev, or beta? They install side-by-side with Chrome quite nicely.

Can you look for crash dumps and attach one to this bug if you find one? Comments #15 and #16 suggest the easiest ways to do this. Comment #13 is a more complicated method if the other two don't work.

The revert (which should fix this) landed here:

https://chromium-review.googlesource.com/934862

That means that the repro will go away soon, probably Saturday morning, so if attaching the debugger to get a crash dump is needed it will have to be soon.

Thank you for your patience.


Manually pasting in the fix CL message:

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1155ea4b7b9dea521fe031cdd6b1856ab21dfd8b

commit 1155ea4b7b9dea521fe031cdd6b1856ab21dfd8b
Author: Bruce Dawson <brucedawson@chromium.org>
Date: Fri Feb 23 19:18:14 2018

Revert "Fix Windows sandbox to work with Application Verifier"

This reverts commit da69db99fc30649947ba1f78b48fc9e6afe0de0b.

Reason for revert: Suspected of causing the crashes tracked by  crbug.com/814641 

Original change's description:
> Fix Windows sandbox to work with Application Verifier
> 
> Application Verifier is a useful tool for tracking down bugs on Windows.
> However Chrome's sandbox has not played well with App Verifier - it ends
> up initializing a heap before the process is ready for this. This change
> teaches the NtMapViewOfSection to skip InitHeap when handling the App
> Verifier DLLs, which don't need patching anyway.
> 
> With this change I can run Chrome with the default App Verifier settings
> enabled with the exception of Handles and TLS (those failures probably
> don't represent real bugs), without using --no-sandbox.
> 
> Bug:  752344 
> Change-Id: Id1fd4be9c0f080513bbdb649ed5cf2637df8474c
> Reviewed-on: https://chromium-review.googlesource.com/925821
> Reviewed-by: James Forshaw <forshaw@chromium.org>
> Reviewed-by: Will Harris <wfh@chromium.org>
> Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#538307}

TBR=brucedawson@chromium.org,forshaw@chromium.org,wfh@chromium.org

# Not skipping CQ checks because original CL landed > 1 day ago.

Bug:  752344 
Change-Id: Icd8feea1779f2891389762db0773ecf0ed9ad762
Reviewed-on: https://chromium-review.googlesource.com/934862
Reviewed-by: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Will Harris <wfh@chromium.org>
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Cr-Commit-Position: refs/heads/master@{#538851}
[modify] https://crrev.com/1155ea4b7b9dea521fe031cdd6b1856ab21dfd8b/sandbox/win/src/sandbox_nt_util.cc
[modify] https://crrev.com/1155ea4b7b9dea521fe031cdd6b1856ab21dfd8b/sandbox/win/src/sandbox_nt_util.h
[modify] https://crrev.com/1155ea4b7b9dea521fe031cdd6b1856ab21dfd8b/sandbox/win/src/target_interceptions.cc

dchaubey@ please verify in latest canary.
Labels: TE-Verified-M66 TE-Verified-65.0.3356.0
Update:-
 Re-tested this issue on Windows (7,8) machines using latest Chrome Canary build# 66.0.3356.0 (Official Build) (64-bit) and fix is working as expected.. Hence adding TE-Verified labels. 

Please find the attached screen-cast for reference.

Thanks..!
Latest Canary behavior.mp4
785 KB View Download
Thanks for verifying, Bruce can you please tag the bug as fixed.
Owner: brucedaw...@chromium.org
Status: Fixed (was: Untriaged)
Project Member

Comment 24 by bugdroid1@chromium.org, Apr 23 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a695cf342559b85f2871a5851172fae9d0af3ae3

commit a695cf342559b85f2871a5851172fae9d0af3ae3
Author: Bruce Dawson <brucedawson@chromium.org>
Date: Mon Apr 23 11:46:21 2018

Fix Windows sandbox to work with Application Verifier

Application Verifier is a useful tool for tracking down bugs on Windows.
However Chrome's sandbox has not played well with App Verifier - it ends
up initializing a heap before the process is ready for this. This change
teaches the NtMapViewOfSection to skip InitHeap when handling the App
Verifier DLLs, which don't need patching anyway.

With this change I can run Chrome with the default App Verifier settings
enabled with the exception of Handles and TLS (those failures probably
don't represent real bugs), without using --no-sandbox.

This is an update to crrev.com/c/925821 with null checks added. The
previous CL was reverted due to crashes on some machines, which
unfortunately were never diagnosed.

Bug:  752344 ,792289, 814641 
Change-Id: I52eadcdbc2c1f3dcc76bddf3b97e903143461757
Reviewed-on: https://chromium-review.googlesource.com/1014666
Commit-Queue: Bruce Dawson <brucedawson@chromium.org>
Reviewed-by: Will Harris <wfh@chromium.org>
Cr-Commit-Position: refs/heads/master@{#552672}
[modify] https://crrev.com/a695cf342559b85f2871a5851172fae9d0af3ae3/sandbox/win/src/sandbox_nt_util.cc
[modify] https://crrev.com/a695cf342559b85f2871a5851172fae9d0af3ae3/sandbox/win/src/sandbox_nt_util.h
[modify] https://crrev.com/a695cf342559b85f2871a5851172fae9d0af3ae3/sandbox/win/src/target_interceptions.cc

Labels: TE-Verified-M68 TE-Verified-68.0.3405.0
Update:-
 Re-tested this issue on Windows (7,8) machines using latest Chrome Canary build# 68.0.3405.0 (Official Build) (64-bit) and fix is working as expected.. Hence adding TE-Verified labels. 

Please find the attached screen-cast for reference.

Thanks..!
Latest_Canary_behavior.mp4
814 KB View Download
Thanks for verifying.

If the fix works then that presumably means that we are loading a module with no name quite early in the process startup process. It could still be interesting to know what module this is. If Chrome is run under windbg then this information will be printed out, by default I believe. This command should do the trick:

windbg -g -G -o

If somebody who can reproduce the problem can do this and then save the debugger output to a text file and attach it that would be helpful.

Sign in to add a comment