New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Ill in v8::internal::FullEvacuationVerifier::VerifyPointers

Project Member Reported by ClusterFuzz, Apr 12 2018

Issue description

Detailed report: https://clusterfuzz.com/testcase?key=4606656135299072

Job Type: linux_asan_chrome_mp
Platform Id: linux

Crash Type: Ill
Crash Address: 0x55dfc4c28ea8
Crash State:
  v8::internal::FullEvacuationVerifier::VerifyPointers
  void v8::internal::BodyDescriptorApply<v8::internal::CallIterateBody, void, v8::
  v8::internal::EvacuationVerifier::VerifyEvacuation
  
Sanitizer: address (ASAN)

Regressed: https://clusterfuzz.com/revisions?job=linux_asan_chrome_mp&range=536709:536711

Reproducer Testcase: https://clusterfuzz.com/download?testcase_id=4606656135299072

Issue filed automatically.

See https://github.com/google/clusterfuzz-tools for more information.
 
Labels: Restrict-View-SecurityTeam
Project Member

Comment 2 by ClusterFuzz, Apr 12 2018

Components: Blink>JavaScript>GC
Labels: Test-Predator-Auto-Components
Automatically applying components based on crash stacktrace and information from OWNERS files.

If this is incorrect, please apply the Test-Predator-Wrong-Components label.
Project Member

Comment 3 by ClusterFuzz, Apr 12 2018

Cc: u...@chromium.org cbruni@chromium.org
Labels: Test-Predator-Auto-CC
Automatically adding ccs based on suspected regression changelists:

[heap] Add description parameter to RootVisitor methods. by ulan@chromium.org - https://chromium.googlesource.com/v8/v8/+/bba08b33145952466ebd4b205bfa63ccffd5623f

[errors] Use FATAL macro where possible by cbruni@chromium.org - https://chromium.googlesource.com/v8/v8/+/52b3b491a55f5f3233ca78e0c22c37384b92670e

If this is incorrect, please let us know why and apply the Test-Predator-Wrong-CLs label.
Labels: -Type-Bug Type-Bug-Security
Owner: u...@chromium.org
Status: Assigned (was: Untriaged)

Comment 6 by u...@chromium.org, Apr 13 2018

Owner: ----
Status: Unconfirmed (was: Assigned)
"[heap] Add description parameter to RootVisitor methods." is a pure mechanical CL that adds a parameter to one function. The parameter is not even used. I don't see how it can cause a crash.

I cannot reproduce the crash locally with
/google/data/ro/teams/clusterfuzz-tools/releases/clusterfuzz reproduce 4606656135299072

I tried different combinations of --build download and --disable-xvfb.

> UnreproducibleError: The crash cannot be reproduced after trying 3 times.

lokihardt@, how should we proceed here? Close as not actionable?
I can reproduce the issue on a locally built d8 (release mode without ASan). It seems a garbage collection needs to be triggered at a certain timing which makes it hard to reproduce.

lokihardt@ubuntu:~/v8/out.gn/x64.release$ cat ~/test.js
for (let i = 0; i < 10; i++) {
    let [__v_2] = [,], arr = [...Array(9063)];
    for (__v_2 = 0; __v_2 < 400; __v_2++) {
        Reflect.ownKeys(arr).shift();
        Array(64386);
    }
}

lokihardt@ubuntu:~/v8/out.gn/x64.release$ ./d8 -v
V8 version 6.7.0 (candidate)
d8> ^C
lokihardt@ubuntu:~/v8/out.gn/x64.release$ ./d8 ~/test.js
Received signal 11 SEGV_MAPERR 7ef82e680000

==== C stack trace ===============================

 [0x555555e5ba91]
 [0x5555577d49cf]
 [0x7ffff776a390]
 [0x555556ae6f07]
 [0x555556aea6f0]
 [0x555556af78ca]
 [0x555556ae554c]
 [0x555556a32fad]
 [0x555556a5b98d]
 [0x5555577eb29e]
 [0x5555577cdece]
 [0x7ffff77606ba]
 [0x7ffff6d6f3dd]
[end of stack trace]
Segmentation fault

Comment 8 by u...@chromium.org, Apr 13 2018

Awesome! Thanks a lot. I'll try to repro

Comment 9 by u...@chromium.org, Apr 13 2018

Owner: u...@chromium.org
Status: Assigned (was: Unconfirmed)
I can reproduce locally. Debugging.

#0  0x0000555556b8a547 in CopyWords<v8::internal::Object*> () at ../../src/utils.h:1136
#1  CopyBlock () at ../../src/heap/heap-inl.h:400
#2  MigrateObject () at ../../src/heap/scavenger-inl.h:50
#3  PromoteObject () at ../../src/heap/scavenger-inl.h:109
#4  EvacuateObjectDefault () at ../../src/heap/scavenger-inl.h:137
#5  EvacuateObject () at ../../src/heap/scavenger-inl.h:224
#6  ScavengeObject () at ../../src/heap/scavenger-inl.h:253
#7  0x0000555556b8dd30 in HandleSlot () at ../../src/heap/scavenger.cc:56
#8  0x0000555556b9aeaa in VisitPointers () at ../../src/heap/scavenger.cc:30
#9  IteratePointers<v8::internal::IterateAndScavengePromotedObjectsVisitor> () at ../../src/objects-body-descriptors-inl.h:65
#10 IterateBody<v8::internal::IterateAndScavengePromotedObjectsVisitor> () at ../../src/objects-body-descriptors.h:103
#11 apply<v8::internal::FlexibleBodyDescriptor<16>, v8::internal::IterateAndScavengePromotedObjectsVisitor> () at ../../src/objects-body-descriptors-inl.h:722
#12 BodyDescriptorApply<v8::internal::CallIterateBody, void, v8::internal::Map*, v8::internal::HeapObject*, int, v8::internal::IterateAndScavengePromotedObjectsVisitor*> ()
    at ../../src/objects-body-descriptors-inl.h:566
#13 0x0000555556b88c3c in IterateBodyFast<v8::internal::IterateAndScavengePromotedObjectsVisitor> () at ../../src/objects-body-descriptors-inl.h:728
#14 IterateAndScavengePromotedObject () at ../../src/heap/scavenger.cc:106
#15 Process () at ../../src/heap/scavenger.cc:166
#16 0x0000555556ac1f8d in RunInParallel () at ../../src/heap/heap.cc:2031
#17 0x0000555556af4cbd in RunInternal () at ../../src/heap/item-parallel-job.cc:44
#18 0x0000555556af5da0 in Run () at ../../src/cancelable-task.h:148
#19 Run () at ../../src/heap/item-parallel-job.cc:117
#20 0x0000555556a9c9d1 in Scavenge () at ../../src/heap/heap.cc:2147
#21 0x0000555556a91b54 in PerformGarbageCollection () at ../../src/heap/heap.cc:1727
#22 0x0000555556a8cff3 in CollectGarbage () at ../../src/heap/heap.cc:1374
#23 0x0000555556aaff5c in AllocateRawWithRetry () at ../../src/heap/heap.cc:4488
#24 0x0000555556a21cf1 in AllocateRawArray () at ../../src/heap/factory.cc:73
#25 AllocateRawFixedArray () at ../../src/heap/factory.cc:86
#26 NewFixedArrayWithFiller () at ../../src/heap/factory.cc:200
#27 0x0000555556dea90f in Allocate () at ../../src/objects.cc:18205
#28 Rehash () at ../../src/objects.cc:18339
#29 0x0000555556ed1212 in EnsureGrowable () at ../../src/objects.cc:18230
#30 Add () at ../../src/objects.cc:18282
#31 0x0000555556d505ae in AddKey () at ../../src/keys.cc:80
#32 0x00005555569006c4 in CollectElementIndicesImpl () at ../../src/elements.cc:1159
#33 CollectElementIndices () at ../../src/elements.cc:1144
#34 0x0000555556d56f81 in CollectElementIndices () at ../../src/elements.h:92
#35 CollectOwnElementIndices () at ../../src/keys.cc:561
#36 0x0000555556d53cf1 in CollectOwnKeys () at ../../src/keys.cc:743
#37 0x0000555556d51780 in CollectKeys () at ../../src/keys.cc:178
#38 0x0000555556d5576e in GetKeysSlow () at ../../src/keys.cc:451
#39 0x0000555556d4fc64 in GetKeys () at ../../src/keys.cc:385
#40 GetKeys () at ../../src/keys.cc:42

Labels: M-66 Security_Severity-Medium Security_Impact-Stable

Comment 11 by u...@chromium.org, Apr 14 2018

Cc: bmeu...@chromium.org jgruber@chromium.org
Bisect the reliable repro from comment #7 to
"[builtins] Perform stack check on entry of all TFJ builtins"
https://chromium-review.googlesource.com/c/v8/v8/+/867237

M65 is also affected.

Looks like one of the TFJ builtins is called from a place that is not prepared for GC.

jgruber@, bmeurer@, is any call to TFJ considered a normal call forcing TF to emit pointers maps and write barrier properly? 
Here's my analysis. I hope it's correct.

1. When rehashing a hash table, it stores the pointer to the new table into the old table (https://cs.chromium.org/chromium/src/v8/src/objects.cc?rcl=a2ca1996873f3ffa79d9495fb2cf4e7c0e51d9e9&l=18369). The new table is directly used as the backing store of the result array of "Reflect.ownKeys(arr)".
2. The shift method invokes the MoveElements method which invokes the Heap::LeftTrimFixedArray method when "len > JSArray::kMaxCopyElements" is fulfilled (https://cs.chromium.org/chromium/src/v8/src/elements.cc?rcl=a2ca1996873f3ffa79d9495fb2cf4e7c0e51d9e9&l=2287).
3. The Heap::LeftTrimFixedArray method cuts out the left 8 bytes (a pointer) of the backing store, and leaves a filter object (8 bytes) in there (https://cs.chromium.org/chromium/src/v8/src/heap/heap.cc?rcl=a2ca1996873f3ffa79d9495fb2cf4e7c0e51d9e9&l=2816). It seems this operation is performed based on the assumption that the pointer to the backing store is not held in other places. But the pointer to it was stored into the old table at 1.
4. So the filter object created at 3 is collected from the old table when a garbage collection triggered, since the size of the filter object is 8 bytes, it hits the assertion (https://cs.chromium.org/chromium/src/v8/src/utils.h?rcl=a2ca1996873f3ffa79d9495fb2cf4e7c0e51d9e9&l=1127).


Comment 13 by u...@chromium.org, Apr 14 2018

Owner: cbruni@chromium.org
Amazing investigation, lokihardt@! I think your explanation is correct.

It matches with the symptoms I saw while debugging: backing store of a hash table pointing to a filler object.

LeftTrimming indeed relies on the assumption that the backing store is pointed to only by the owner.

The problem seems to be that Reflect.ownKeys returns the backing store directly instead of creating a new array. This CL changed that part: https://codereview.chromium.org/2014523002
"Reland of [keys] Simplify KeyAccumulator"

Assigning to Camillo to partially revert that CL or come up with a fix.
Thanks ulan@ for the investigation, and awesome catch lokihardt@!
I'll try to look into this tomorrow/Monday.
Ulan, just to clarify, is #11 an invalid bisect or is there also an issue related to TFJ builtin stack checks?

Comment 16 by u...@chromium.org, Apr 16 2018

Looks like #11 changed that GC timing to make repro more reliable. It seems unrelated.
Status: Started (was: Assigned)
Labels: M-65
Project Member

Comment 19 by bugdroid1@chromium.org, Apr 16 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/7bb79b96bdd29c41acc8cf36c428dd66308e5b66

commit 7bb79b96bdd29c41acc8cf36c428dd66308e5b66
Author: Camillo Bruni <cbruni@chromium.org>
Date: Mon Apr 16 21:07:06 2018

[keys] Don't keep chain of OrderedHashSets in KeyAccumulator

Bug:  chromium:831984 
Change-Id: Ie13b22bc2491acc255557ba0325d8d53c22d6acb
Reviewed-on: https://chromium-review.googlesource.com/1012874
Reviewed-by: Igor Sheludko <ishell@chromium.org>
Commit-Queue: Camillo Bruni <cbruni@chromium.org>
Cr-Commit-Position: refs/heads/master@{#52630}
[modify] https://crrev.com/7bb79b96bdd29c41acc8cf36c428dd66308e5b66/src/keys.cc
[add] https://crrev.com/7bb79b96bdd29c41acc8cf36c428dd66308e5b66/test/mjsunit/regress/regress-crbug-831984.js

Labels: Merge-Request-67 Merge-Request-66
Project Member

Comment 21 by sheriffbot@chromium.org, Apr 16 2018

Labels: -Merge-Request-66 Merge-Review-66 Hotlist-Merge-Review
This bug requires manual review: We are only 0 days from stable.
Please contact the milestone owner if you have questions.
Owners: cmasso@(Android), cmasso@(iOS), josafat@(ChromeOS), abdulsyed@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 22 by sheriffbot@chromium.org, Apr 17 2018

Status: Fixed (was: Started)
Please mark security bugs as fixed as soon as the fix lands, and before requesting merges. This update is based on the merge- labels applied to this issue. Please reopen if this update was incorrect.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 23 by sheriffbot@chromium.org, Apr 17 2018

Labels: -Merge-Request-67 Merge-Approved-67 Hotlist-Merge-Approved
Your change meets the bar and is auto-approved for M67. Please go ahead and merge the CL to branch 3396 manually. Please contact milestone owner if you have questions.
Owners: cmasso@(Android), cmasso@(iOS), kbleicher@(ChromeOS), govind@(Desktop)

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Pls merge your change to M67 branch 3396 ASAP. Thank you.
Project Member

Comment 25 by sheriffbot@chromium.org, Apr 18 2018

Labels: -Restrict-View-SecurityTeam Restrict-View-SecurityNotify
Pls merge your change to M67 branch 3396 ASAP so we can pick it up for next M67 Dev/Beta release.

If already merged to M67 and nothing is pending, pls remove "Merge=Approved-67" label. Thank you.
Project Member

Comment 27 by bugdroid1@chromium.org, Apr 19 2018

Labels: merge-merged-6.7
The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/c4e904f4a682ed9bee33a818f32a177992c0c7d0

commit c4e904f4a682ed9bee33a818f32a177992c0c7d0
Author: Camillo Bruni <cbruni@chromium.org>
Date: Thu Apr 19 00:41:43 2018

Merged: [keys] Don't keep chain of OrderedHashSets in KeyAccumulator

Revision: 7bb79b96bdd29c41acc8cf36c428dd66308e5b66

BUG= chromium:831984 
LOG=N
NOTRY=true
NOPRESUBMIT=true
NOTREECHECKS=true
R=gsathya@chromium.org

Change-Id: I645a52e81c31397494e255d4b30f60a693fde56c
Reviewed-on: https://chromium-review.googlesource.com/1018460
Reviewed-by: Sathya Gunasekaran <gsathya@chromium.org>
Cr-Commit-Position: refs/branch-heads/6.7@{#24}
Cr-Branched-From: 8457e810efd34381448d51d93f50079cf1f6a812-refs/heads/6.7.288@{#2}
Cr-Branched-From: e921be5c4f2c6407936bde750992dedbf47c1016-refs/heads/master@{#52547}
[modify] https://crrev.com/c4e904f4a682ed9bee33a818f32a177992c0c7d0/src/keys.cc
[add] https://crrev.com/c4e904f4a682ed9bee33a818f32a177992c0c7d0/test/mjsunit/regress/regress-crbug-831984.js

Project Member

Comment 28 by ClusterFuzz, Apr 19 2018

ClusterFuzz has detected this issue as fixed in range 551286:551287.

Detailed report: https://clusterfuzz.com/testcase?key=4606656135299072

Job Type: linux_asan_chrome_mp
Platform Id: linux

Crash Type: Ill
Crash Address: 0x55dfc4c28ea8
Crash State:
  v8::internal::FullEvacuationVerifier::VerifyPointers
  void v8::internal::BodyDescriptorApply<v8::internal::CallIterateBody, void, v8::
  v8::internal::EvacuationVerifier::VerifyEvacuation
  
Sanitizer: address (ASAN)

Regressed: https://clusterfuzz.com/revisions?job=linux_asan_chrome_mp&range=536709:536711
Fixed: https://clusterfuzz.com/revisions?job=linux_asan_chrome_mp&range=551286:551287

Reproducer Testcase: https://clusterfuzz.com/download?testcase_id=4606656135299072

See https://github.com/google/clusterfuzz-tools for more information.

If you suspect that the result above is incorrect, try re-doing that job on the test case report page.
Project Member

Comment 29 by ClusterFuzz, Apr 19 2018

Labels: ClusterFuzz-Verified
Status: Verified (was: Fixed)
ClusterFuzz testcase 4606656135299072 is verified as fixed, so closing issue as verified.

If this is incorrect, please add ClusterFuzz-Wrong label and re-open the issue.
Labels: -Merge-Approved-67
Labels: -Merge-Review-66 Merge-Approved-66
Please wait a few more days to get more Canary coverage before merge.
Project Member

Comment 32 by sheriffbot@chromium.org, Apr 23 2018

Cc: hablich@chromium.org
This issue has been approved for a merge. Please merge the fix to any appropriate branches as soon as possible!

If all merges have been completed, please remove any remaining Merge-Approved labels from this issue.

Thanks for your time! To disable nags, add the Disable-Nags label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
Project Member

Comment 33 by bugdroid1@chromium.org, Apr 25 2018

Labels: merge-merged-6.6
The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/cb12999ecc3e2875ae7cab317d6c70b10cde7f07

commit cb12999ecc3e2875ae7cab317d6c70b10cde7f07
Author: Camillo Bruni <cbruni@chromium.org>
Date: Wed Apr 25 16:50:46 2018

Merged: [keys] Don't keep chain of OrderedHashSets in KeyAccumulator

Revision: 7bb79b96bdd29c41acc8cf36c428dd66308e5b66

BUG= chromium:831984 
LOG=N
NOTRY=true
NOPRESUBMIT=true
NOTREECHECKS=true
R=ishell@chromium.org

Change-Id: Iba6a94ea2b96627311d1cd3685e3dbd1bcaf341e
Reviewed-on: https://chromium-review.googlesource.com/1028233
Reviewed-by: Igor Sheludko <ishell@chromium.org>
Cr-Commit-Position: refs/branch-heads/6.6@{#51}
Cr-Branched-From: d500271571b92cb18dcd7b15885b51e8f437d640-refs/heads/6.6.346@{#1}
Cr-Branched-From: 265ef0b635f8761df7c89eb4e8ec9c1a6ebee184-refs/heads/master@{#51624}
[modify] https://crrev.com/cb12999ecc3e2875ae7cab317d6c70b10cde7f07/src/keys.cc
[add] https://crrev.com/cb12999ecc3e2875ae7cab317d6c70b10cde7f07/test/mjsunit/regress/regress-crbug-831984.js

Labels: -Merge-Approved-66
Labels: Release-1-M66
Labels: NodeJS-Backport-Rejected
Not needed for active Node.js release lines.
Labels: Hotlist-Torque
Cc: tebbi@chromium.org
Cc: jarin@chromium.org
Project Member

Comment 40 by sheriffbot@chromium.org, Jul 25

Labels: -Restrict-View-SecurityNotify allpublic
This bug has been closed for more than 14 weeks. Removing security view restrictions.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot

Sign in to add a comment