From logs:

#
# Fatal error in ../../v8/src/heap/spaces-inl.h, line 468
# Debug check failed: Page::FromAddress(top()) == Page::FromAddress(top_on_previous_step_) (0x16ba80000 vs. 0x16bb00000).
#
0   Chromium Framework                  0x0000000111cfcfbc base::debug::StackTrace::StackTrace(unsigned long) + 28
1   Chromium Framework                  0x000000011c669664 gin::(anonymous namespace)::PrintStackTrace() + 180
2   Chromium Framework                  0x000000011be54b16 V8_Fatal(char const*, int, char const*, ...) + 406
3   Chromium Framework                  0x000000011be5442f v8::base::(anonymous namespace)::DefaultDcheckHandler(char const*, int, char const*) + 47
4   Chromium Framework                  0x000000010f557959 v8::internal::NewSpace::AllocateRaw(int, v8::internal::AllocationAlignment) + 1017
5   Chromium Framework                  0x000000010f556e08 v8::internal::Heap::AllocateRaw(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) + 664
6   Chromium Framework                  0x000000010f5f6c73 v8::internal::Heap::AllocateFillerObject(int, bool, v8::internal::AllocationSpace) + 35
7   Chromium Framework                  0x000000010f500e86 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) + 38
8   Chromium Framework                  0x000000010ff4488d v8::internal::__RT_impl_Runtime_AllocateInNewSpace(v8::internal::Arguments, v8::internal::Isolate*) + 429
9   Chromium Framework                  0x000000010ff43e15 v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) + 805

Ulan, Ali: Please revert the CL(s) dealing with the allocation observer code. (I have not followed the progress there.)

What is the problem with this feature? Why is it hard to test and/or get the invariants correct? We should figure this out before relanding to avoid the problems with embedders.

 Issue 791454  has been merged into this issue.

mlippautz: While the assert has been useful in understanding where the assumed invariants don't hold, I am removing it for now in this CL: https://chromium-review.googlesource.com/c/v8/v8/+/806516.

At this point the assert is getting in the way of simplifying the code paths significantly. Sampling heap profiler is 'informational' so the assert can be removed and added back after the refactoring/cleanup/clear establishment of invariants.

As for why it has been difficult: because the inline allocation code paths are complex and don't share logic between NewSpace and OldSpace. Certain assumed invariants don't hold (e.g. linear allocator doesn't necessarily allocate linearly). If there were clear invariants for inline allocation and a choke point between spaces, this would have been a lot easier. This is want I intend to by adding a common base class for NewSpace and OldSpace to refactor the inline allocation code. The assert has been useful so far, but I think I have a pretty good idea how to proceed, and the assert can be disabled for now.

Note that this is seen on non-ASAN bots and on Windows as well:

https://ci.chromium.org/buildbot/tryserver.chromium.angle/win_angle_rel_ng/8299
https://ci.chromium.org/buildbot/tryserver.chromium.angle/win_angle_rel_ng/8304

Representative stack trace:

  	Last event: d68.fac: Access violation - code c0000005 (first/second chance not available)
  	  debugger time: Tue Dec  5 00:17:40.356 2017 (UTC - 8:00)
  	ChildEBP RetAddr  Args to Child              
  	0048d774 67fb8f6a 00000016 0048d798 67fb7185 chrome_child!base::win::SetAbortBehaviorForCrashReporting+0x20
  	0048d780 67fb7185 0048d7ac 01336740 0602f138 chrome_child!v8::base::OS::Abort+0xa
  	0048d798 67fb6ff6 69c1a03b 000001d4 6a0b5a90 chrome_child!V8_Fatal+0x85
  	0048d7b0 66869d4e 69c1a03b 000001d4 05f53d18 chrome_child!v8::base::SetPrintStackTrace+0x26
  	0048d7d4 66869a04 0048d7e8 00000028 00000000 chrome_child!v8::internal::NewSpace::AllocateRaw+0x1ee
  	0048d7fc 6689abfd 0048d848 00000028 00000000 chrome_child!v8::internal::Heap::AllocateRaw+0x1e4
  	0048d824 668506ef 0048d848 00000028 00000000 chrome_child!v8::internal::Heap::AllocateFillerObject+0x1d
  	0048d85c 66b35ef9 0048d874 00000028 00000000 chrome_child!v8::internal::Factory::NewFillerObject+0x2f
  	0048d89c 66b35c85 01305b68 00000000 40a00e00 chrome_child!v8::internal::Runtime_AllocateInNewSpace+0x339
  	0048d910 3a7862aa 00000001 0048d93c 01305b68 chrome_child!v8::internal::Runtime_AllocateInNewSpace+0xc5

The issue in this case – the invariant that didn't hold – was that Page::FromAddress doesn't return the correct page when we are at the boundary. The right thing to use was Page::FromAllocationAreaAddress.

Ulan already fixed this (https://chromium.googlesource.com/v8/v8.git/+/7f8f2833663f77d41cdf95b0e2a2a64e844aac7d%5E%21/src/heap/spaces-inl.h) a while ago bug missed one of assertions.

When do we expect the fix to roll into Chromium?  We're seeing this assertion failure across most of the GPU FYI bots now.

Current GC sheriff is out sick.

Secondary is hpayer. Hannes could you ensure the problematic dcheck is removed asap?

https://chromium-review.googlesource.com/c/v8/v8/+/806516 should land imminently.

Thanks for pushing the fix through!

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/00a77a9f4aeed0a2dd7124e564991e59a9f393c6

commit 00a77a9f4aeed0a2dd7124e564991e59a9f393c6
Author: Ali Ijaz Sheikh <ofrobots@google.com>
Date: Tue Dec 05 16:08:23 2017

[heap] Fix top_on_previous_step_ check in NewSpace::AllocateRaw.

See also: https://chromium-review.googlesource.com/c/v8/v8/+/738204

BUG= chromium:791582 

Change-Id: Ife3acf35eeaa6fdebd5ea2fabc1678ec762b3ed3
Reviewed-on: https://chromium-review.googlesource.com/806516
Reviewed-by: Ulan Degenbaev <ulan@chromium.org>
Commit-Queue: Ali Ijaz Sheikh <ofrobots@google.com>
Cr-Commit-Position: refs/heads/master@{#49873}
[modify] https://crrev.com/00a77a9f4aeed0a2dd7124e564991e59a9f393c6/src/heap/spaces-inl.h

> Ulan already fixed this (https://chromium.googlesource.com/v8/v8.git/+/7f8f2833663f77d41cdf95b0e2a2a64e844aac7d%5E%21/src/heap/spaces-inl.h)
> a while ago bug missed one of assertions.
Small nit: my fix was complete, this assertion wasn't there at that time.

Crashes are still affecting current tryjobs:

https://ci.chromium.org/buildbot/tryserver.chromium.mac/mac_optional_gpu_tests_rel/18429

Did this fix not yet roll in for this tryjob or is there another bug?

  	Thread 0 (crashed)
  	 0  Chromium Framework!__ZN2v84base2OS5AbortEv + 0x12
  	    rax = 0x0000000000000000   rdx = 0x0000000000012068
  	    rcx = 0x0000040000000503   rbx = 0x0000000112b2b6e2
  	    rsi = 0x0000000000091600   rdi = 0x00007fffa8930048
  	    rbp = 0x00007fff58b702d0   rsp = 0x00007fff58b701d8
  	     r8 = 0x0000000000000040    r9 = 0x00007fffa8930040
  	    r10 = 0xffffffffffffffff   r11 = 0x0000000000012068
  	    r12 = 0x00007fffa8930a20   r13 = 0x00007fc58280da20
  	    r14 = 0x0000000112f517f2   r15 = 0x00000000000001d4
  	    rip = 0x000000011035a8d2
  	    Found by: given as instruction pointer in context
  	 1  Chromium Framework!__ZN2v84base12_GLOBAL__N_120DefaultDcheckHandlerEPKciS3_ + 0x15
  	    rbp = 0x00007fff58b702e0   rsp = 0x00007fff58b702e0
  	    rip = 0x00000001103559b5
  	    Found by: previous frame's frame pointer
  	 2  Chromium Framework!__ZN2v88internal8NewSpace11AllocateRawEiNS0_19AllocationAlignmentE + 0x1eb
  	    rbp = 0x00007fff58b70310   rsp = 0x00007fff58b702f0
  	    rip = 0x000000010b444a9b
  	    Found by: previous frame's frame pointer
  	 3  Chromium Framework!__ZN2v88internal4Heap11AllocateRawEiNS0_15AllocationSpaceENS0_19AllocationAlignmentE + 0xf3
  	    rbp = 0x00007fff58b70350   rsp = 0x00007fff58b70320
  	    rip = 0x000000010b444573
  	    Found by: previous frame's frame pointer
  	 4  Chromium Framework!__ZN2v88internal4Heap20AllocateFillerObjectEibNS0_15AllocationSpaceE + 0x23
  	    rbp = 0x00007fff58b70390   rsp = 0x00007fff58b70360
  	    rip = 0x000000010b48af13
  	    Found by: previous frame's frame pointer
  	 5  Chromium Framework!__ZN2v88internal7Factory15NewFillerObjectEibNS0_15AllocationSpaceE + 0x25
  	    rbp = 0x00007fff58b703d0   rsp = 0x00007fff58b703a0
  	    rip = 0x000000010b41d825
  	    Found by: previous frame's frame pointer
  	 6  Chromium Framework!__ZN2v88internalL36__RT_impl_Runtime_AllocateInNewSpaceENS0_9ArgumentsEPNS0_7IsolateE + 0x7c
  	    rbp = 0x00007fff58b70420   rsp = 0x00007fff58b703e0
  	    rip = 0x000000010b80871c
  	    Found by: previous frame's frame pointer
  	 7  0xdcbe1f84264
  	    rbp = 0x00007fff58b70450   rsp = 0x00007fff58b70430
  	    rip = 0x00000dcbe1f84264
  	    Found by: previous frame's frame pointer
  	 8  0xdcbe1f948d1
  	    rbp = 0x00007fff58b704a0   rsp = 0x00007fff58b70460
  	    rip = 0x00000dcbe1f948d1
  	    Found by: previous frame's frame pointer
  	 9  0xdcbe20eefcc
  	    rbp = 0x00007fff58b704c0   rsp = 0x00007fff58b704b0
  	    rip = 0x00000dcbe20eefcc
  	    Found by: previous frame's frame pointer
  	10  0xdcbe1f9a95a
  	    rbp = 0x00007fff58b70578   rsp = 0x00007fff58b704d0
  	    rip = 0x00000dcbe1f9a95a
  	    Found by: previous frame's frame pointer

Based on the assertion failure in that test:

"# Debug check failed: Page::FromAddress(top()) == Page::FromAddress(top_on_previous_step_) (0x192fc6f00000 vs. 0x192fc6f80000)."

I don't think the fix has rolled in yet.

It's difficult to see the status of the V8 auto-roller. Filed Issue v8:7169 about that.

The fix has indeed not rolled in yet, and the list of CLs waiting to roll in is large at this point. It's not clear to me whether a change on the V8 side has broken automatic rolls. There was certainly breakage on the Chromium tree (compile failures) which broke some of the try jobs.

Fix rolled here: https://chromium-review.googlesource.com/c/chromium/src/+/809911

The CQ cycle was long due to some other breakages (>2 hours). And there was a previous cherry pick roll in the CQ as well https://chromium-review.googlesource.com/c/chromium/src/+/809193 which was >4 hours in CQ due to yet another infra breakage :(. So our cycle was a bit over 6 hours sadly. I missed to add this bug's fix in yesterday's cherry-pick for  issue 792105 . I think  I made the cherry-pick right before reading this bug and then forgot about it :(

ClusterFuzz testcase 6194866395807744 is verified as fixed, so closing issue as verified.

If this is incorrect, please add ClusterFuzz-Wrong label and re-open the issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/ac5b4223adf2152e80f4b410011af966ebb8e4e5

commit ac5b4223adf2152e80f4b410011af966ebb8e4e5
Author: Ulan Degenbaev <ulan@chromium.org>
Date: Wed Dec 06 16:19:26 2017

[heap] Add regression test for 791582.

Bug:  chromium:791582 
Change-Id: Ic2b4289431a4bd7b4b5a37437d25ebccd493497a
Reviewed-on: https://chromium-review.googlesource.com/809130
Reviewed-by: Michael Lippautz <mlippautz@chromium.org>
Commit-Queue: Ulan Degenbaev <ulan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#49903}
[modify] https://crrev.com/ac5b4223adf2152e80f4b410011af966ebb8e4e5/test/cctest/heap/heap-tester.h
[modify] https://crrev.com/ac5b4223adf2152e80f4b410011af966ebb8e4e5/test/cctest/heap/test-spaces.cc