New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 718859 link

Starred by 13 users

Issue metadata

Status: Fixed
Owner:
Closed: May 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Mac
Pri: 1
Type: Bug



Sign in to add a comment

Mac Heap Corruption

Project Member Reported by cbruni@chromium.org, May 5 2017

Issue description

We're seeing rather high numbers of heap corruption on Mac since stable M57.0.2987.133.
Currently we're unsure of the rootcause(s) for the crashers.

What stands out are the crashers on http://www.yelp.com/, a page that usually doesn't get much attention.


Version distribution for mac crashers on yelp: https://crash.corp.google.com/browse?q=product.name%3D%27Chrome_Mac%27%20AND%20custom_data.ChromeCrashProto.channel%3D%27%27%20AND%20custom_data.ChromeCrashProto.ptype%3D%27renderer%27%20%20AND%20custom_data.ChromeCrashProto.url.simplified%3D%27https%3A%2F%2Fwww.yelp.com%2F*%27&ignore_case=false&enable_rewrite=true&omit_field_name=&omit_field_value=&omit_field_opt=%3D

58.0.3029.96	5.13%	687	
58.0.3029.81	28.18%	3771	
57.0.2987.133	60.61%	8110	
57.0.2987.110	3.47%	464	
57.0.2987.98	0.65%	87	


M57 and M58 show the same top 5 signatures with roughly 1 crasher per client each:

1  v8::internal::MarkCompactCollector::RecordRelocSlot
2  v8::internal::JSObject::MigrateSlowToFastStack
3  Memory corruption (v8)
4  v8::internal::Runtime_CreateRegExpLiteral
5  v8::internal::IC::UpdateState

Uptime is between 4 and 10 mins.

Per-day distribution is NOT correlated to chrome release:
1	2017/05/05	3.72%	501	
2	2017/05/04	12.50%	1681	
3	2017/05/03	11.45%	1540	
4	2017/05/02	12.66%	1703	
5	2017/05/01	13.54%	1822	
6	2017/04/30	9.69%	1304	
7	2017/04/29	9.71%	1306	
8	2017/04/28	11.61%	1562	
9	2017/04/27	12.38%	1665	
10	2017/04/26	0.48%	65	
11	2017/04/25	0.01%	1	
12	2017/04/24	0.01%	1	

Chrome release graphs: 
https://omahadashwiki.corp.google.com/generate?active_days=1&app=%7B8A69D345-D564-463C-AFF1-A69D9E530F96%7D&app_pretty=%7B8A69D345-D564-463C-AFF1-A69D9E530F96%7D&breakout=app_version&bv=*&completion_ratio=0.999&count=unique&data_category=installs&display_name=Installs+by+Version+%28120+days%29&formula=Sum%28%29&formula_name=total&limit=10&num_days=120&start_day=*&style=fancy&width=1400&height=650
 

Comment 1 by danno@chromium.org, May 7 2017

Owner: jkummerow@chromium.org
Status: Assigned (was: Untriaged)
My understanding is that there is a collective effort to look at this from different angles across the V8 team, but to get this out of the triage queue, I'm giving it somewhat arbitrarily to Jakob. Please feel free to re-assign if you think there is a more appropriate owner.
Project Member

Comment 2 by bugdroid1@chromium.org, May 10 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/6bfee50e15f1dda1bcc164d969d2c94453aa04dd

commit 6bfee50e15f1dda1bcc164d969d2c94453aa04dd
Author: jkummerow <jkummerow@chromium.org>
Date: Wed May 10 12:54:01 2017

[deserializer] Make large object deserialization GC safe

When black allocation is turned on at deserialization time, then
slots in deserialized objects have to be visited by the incremental
marker. For spaces with reservations, this has always been done; for
large object space with its special handling, this patch adds it.

Additionally, we must ensure that no incremental steps that might
cause incremental marking to finish are performed while there is an
AlwaysAllocateScope around.

BUG= chromium:718859 

Review-Url: https://codereview.chromium.org/2868103002
Cr-Commit-Position: refs/heads/master@{#45231}

[modify] https://crrev.com/6bfee50e15f1dda1bcc164d969d2c94453aa04dd/src/heap/heap.cc
[modify] https://crrev.com/6bfee50e15f1dda1bcc164d969d2c94453aa04dd/src/heap/heap.h
[modify] https://crrev.com/6bfee50e15f1dda1bcc164d969d2c94453aa04dd/src/heap/incremental-marking.cc
[modify] https://crrev.com/6bfee50e15f1dda1bcc164d969d2c94453aa04dd/src/snapshot/deserializer.cc
[modify] https://crrev.com/6bfee50e15f1dda1bcc164d969d2c94453aa04dd/test/cctest/test-serialize.cc

Labels: Merge-Request-58 Merge-Request-59
This crash has never happened much on Canary, so data from that channel is inconclusive, but at least it seems that nothing got worse. Let's get the fix out to more affected users.
Project Member

Comment 4 by sheriffbot@chromium.org, May 15 2017

Labels: -Merge-Request-59 Hotlist-Merge-Approved Merge-Approved-59
Your change meets the bar and is auto-approved for M59. Please go ahead and merge the CL to branch 3071 manually. Please contact milestone owner if you have questions.
Owners: amineer@(Android), cmasso@(iOS), gkihumba@(ChromeOS), Abdul Syed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Comment 5 by jochen@chromium.org, May 15 2017

is this bad enough to warrant another stable push?
#5: I don't think we have to do another stable push for this. However, if we do end up spinning another build, I think it would be nice to pick up this fix; hence my offer to back-merge it.
Project Member

Comment 7 by bugdroid1@chromium.org, May 15 2017

Labels: merge-merged-5.9
The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/802a4c76b0c407d6ca225f587181f3a99eaa78db

commit 802a4c76b0c407d6ca225f587181f3a99eaa78db
Author: Jakob Kummerow <jkummerow@chromium.org>
Date: Mon May 15 11:33:22 2017

Merged: [deserializer] Make large object deserialization GC safe

Revision: 6bfee50e15f1dda1bcc164d969d2c94453aa04dd

BUG= chromium:718859 
LOG=N
NOTRY=true
NOPRESUBMIT=true
NOTREECHECKS=true
R=ulan@chromium.org

Change-Id: I724a2a84893150c820dd3d9c78bff7d86067f3dc
Reviewed-on: https://chromium-review.googlesource.com/505609
Reviewed-by: Ulan Degenbaev <ulan@chromium.org>
Cr-Commit-Position: refs/branch-heads/5.9@{#45}
Cr-Branched-From: fe9bb7e6e251159852770160cfb21dad3cf03523-refs/heads/5.9.211@{#1}
Cr-Branched-From: 70ad23791a21c0dd7ecef8d4d8dd30ff6fc291f6-refs/heads/master@{#44591}
[modify] https://crrev.com/802a4c76b0c407d6ca225f587181f3a99eaa78db/src/heap/heap.cc
[modify] https://crrev.com/802a4c76b0c407d6ca225f587181f3a99eaa78db/src/heap/heap.h
[modify] https://crrev.com/802a4c76b0c407d6ca225f587181f3a99eaa78db/src/heap/incremental-marking.cc
[modify] https://crrev.com/802a4c76b0c407d6ca225f587181f3a99eaa78db/src/snapshot/deserializer.cc
[modify] https://crrev.com/802a4c76b0c407d6ca225f587181f3a99eaa78db/test/cctest/test-serialize.cc

Comment 8 by jochen@chromium.org, May 15 2017

carrying forward the rationale from https://bugs.chromium.org/p/chromium/issues/detail?id=715582#c22 I'd say we don't merge to 5.8
Labels: -Merge-Approved-59
Merge to 59 done.

Leaving the question of whether we should merge to 58 to TPMs. Background: the issue addressed here is causing a variety of corruption-related crash signatures in renderer+extension processes, most visible on Mac, where we estimate that 25% or more of all crashes on those processes on M58 are due to this bug. Since this didn't block the M58 release, it probably isn't enough to warrant a respin either; but if we do push out another release, picking up this crash fix would be good.
Issue 722778 has been merged into this issue.
Labels: -Merge-Request-58 Merge-Rejected-58
We're are not planning any further M58 stable releases. Rejecting merge to M58.
Issue 704298 has been merged into this issue.
Issue 724483 has been merged into this issue.

Comment 14 Deleted

Issue 699431 has been merged into this issue.
Issue 714602 has been merged into this issue.
Issue 705341 has been merged into this issue.
Issue 704558 has been merged into this issue.
Issue 704337 has been merged into this issue.
Issue 701409 has been merged into this issue.
Issue 710395 has been merged into this issue.
Status: Fixed (was: Assigned)
For the record: The fix for this issue is in:
60.0.3098.0 and later M60 builds
59.0.3071.61 and later M59 builds

M58 merge was rejected per #11. Nothing more to be done here.
Issue 719458 has been merged into this issue.
Issue 714603 has been merged into this issue.
Issue 728180 has been merged into this issue.
Issue 730311 has been merged into this issue.

Sign in to add a comment