New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 684568 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: May 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux , Mac
Pri: 2
Type: Bug-Regression



Sign in to add a comment

Crash in LiveObjectIterator::Next making WebGL conformance tests flaky

Project Member Reported by ajuma@chromium.org, Jan 24 2017

Issue description

WebglConformance_conformance_ogles_GL_build_build_041_to_048 flaked on Mac-10.12 Intel GPU, crashing with stack:

Crash reason: EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
Crash address: 0x22ee4000000
Process uptime: 60 seconds

Thread 19 (crashed)
0 Chromium Framework!__ZN2v88internal18LiveObjectIteratorILNS0_23LiveObjectIterationModeE0EE4NextEv + 0xe8
rax = 0x00000000beadbeee rdx = 0x0000000000000002
rcx = 0x0000000000000000 rbx = 0x0000022ee4000000
rsi = 0x00002722382023b9 rdi = 0x0000022ee4000001
rbp = 0x00007000081c9bd0 rsp = 0x00007000081c9ba0
r8 = 0x0000000000000017 r9 = 0x000000000000001f
r10 = 0x000000000000003f r11 = 0x0000000000000002
r12 = 0x0000022ee3f80000 r13 = 0x00007000081c9be0
r14 = 0x00002722382023b9 r15 = 0x0000022ee3fffff8
rip = 0x000000010be69e68
Found by: given as instruction pointer in context
1 Chromium Framework!__ZN2v88internal20MarkCompactCollector7Sweeper8RawSweepEPNS0_4PageENS2_22FreeListRebuildingModeENS2_22FreeSpaceTreatmentModeE + 0x454
rbp = 0x00007000081c9cb0 rsp = 0x00007000081c9be0
rip = 0x000000010be698b4
Found by: previous frame's frame pointer
2 Chromium Framework!__ZN2v88internal20MarkCompactCollector7Sweeper17ParallelSweepPageEPNS0_4PageENS0_15AllocationSpaceE + 0x6d
rbp = 0x00007000081c9cf0 rsp = 0x00007000081c9cc0
rip = 0x000000010be60fdd
Found by: previous frame's frame pointer
3 Chromium Framework!__ZN2v88internal20MarkCompactCollector7Sweeper18ParallelSweepSpaceENS0_15AllocationSpaceEii + 0x1ad
rbp = 0x00007000081c9d50 rsp = 0x00007000081c9d00
rip = 0x000000010be613ed
Found by: previous frame's frame pointer
4 Chromium Framework!__ZN2v88internal20MarkCompactCollector7Sweeper11SweeperTask3RunEv + 0xce
rbp = 0x00007000081c9d80 rsp = 0x00007000081c9d60
rip = 0x000000010be73fae
Found by: previous frame's frame pointer
5 Chromium Framework!__ZN4base12_GLOBAL__N_112WorkerThread10ThreadMainEv + 0x281
rbp = 0x00007000081c9ec0 rsp = 0x00007000081c9d90
rip = 0x000000010d99f301
Found by: previous frame's frame pointer
6 Chromium Framework!__ZN4base12_GLOBAL__N_110ThreadFuncEPv + 0x5f
rbp = 0x00007000081c9ef0 rsp = 0x00007000081c9ed0
rip = 0x000000010d9909ef
Found by: previous frame's frame pointer
7 libsystem_pthread.dylib + 0x3aab
rbp = 0x00007000081c9f10 rsp = 0x00007000081c9f00
rip = 0x00007fffadbfbaab
Found by: previous frame's frame pointer
8 libsystem_pthread.dylib + 0x39f7
rbp = 0x00007000081c9f50 rsp = 0x00007000081c9f20
rip = 0x00007fffadbfb9f7
Found by: previous frame's frame pointer
9 libsystem_pthread.dylib + 0x31fd
rbp = 0x00007000081c9f78 rsp = 0x00007000081c9f60
rip = 0x00007fffadbfb1fd
Found by: previous frame's frame pointer
10 Chromium Framework!__ZN4base14PlatformThread6DetachENS_20PlatformThreadHandleE + 0x70
rsp = 0x00007000081ca028 rip = 0x000000010d990990
Found by: stack scanning

Build:
https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/6576
Swarming task:
https://chromium-swarm.appspot.com/task?id=33e92846863c8d10&refresh=10&show_raw=1

It looks like the most recent change to that code is https://codereview.chromium.org/2637403011 (Reland "[heap] Provide ObjectMarking with marking transitions")

 

Comment 1 by ajuma@chromium.org, Jan 24 2017

Labels: -Pri-2 Pri-1
Status: Untriaged (was: Unconfirmed)
Summary: Crash in LiveObjectIterator::Next making WebGL conformance tests flaky (was: Crash in LiveObjectIterator::Next making WebGL conformance test flaky)
Another flake with the same crash stack: WebglConformance_deqp_functional_gles3_draw_draw_elements on Intel GPU on Mac-10.12, on the Mac 10.10 Release (Intel) GPU fyi bot:
https://build.chromium.org/p/chromium.gpu.fyi/builders/Mac%2010.10%20Release%20%28Intel%29/builds/20973
https://chromium-swarm.appspot.com/task?id=33e9e19ce4f74210&refresh=10&show_raw=1
Cc: u...@chromium.org
Owner: hpayer@chromium.org
Status: Assigned (was: Untriaged)
Assigning to current sheriff. Rotation at 
  https://rotation.googleplex.com/index.html#rotation?id=4838401396178944

The CL should merely be a mechanical refactoring.

Comment 3 by ajuma@chromium.org, Jan 27 2017

Cc: kbr@chromium.org
Components: Internals>GPU>Testing
Labels: OS-Linux
This also caused maps_pixel_test Maps_maps_004 to flake on Linux:
https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/379533
https://chromium-swarm.appspot.com/task?id=33f8635031c80c10&refresh=10&show_raw=1

Comment 4 by kbr@chromium.org, Jan 27 2017

Cc: mvstan...@chromium.org
Labels: -Type-Bug Type-Bug-Regression
The maps_pixel_test flake above looks like it's in a different place in V8's garbage collector:

	Operating system: Linux
	                  0.0.0 Linux 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64
	CPU: amd64
	     family 6 model 94 stepping 3
	     1 CPU
	
	GPU: UNKNOWN
	
	Crash reason:  SIGSEGV
	Crash address: 0x44d3ab80000
	Process uptime: not available
	
	Thread 14 (crashed)
	 0  chrome!Next [atomicops_internals_portable.h : 161 + 0x0]
	    rax = 0x00000000beadbeee   rdx = 0x0000000000000002
	    rcx = 0x0000000000000000   rbx = 0x00007ff2a71a8938
	    rsi = 0x00000422933823b9   rdi = 0x0000044d3ab7fff9
	    rbp = 0x0000044d3ab80000   rsp = 0x00007ff2a71a8870
	     r8 = 0x000000000000003f    r9 = 0x000000000000001f
	    r10 = 0x0000000000000002   r11 = 0x0000000000000206
	    r12 = 0x00000422933823b9   r13 = 0x0000044d3ab80001
	    r14 = 0x0000044d3ab7fff8   r15 = 0x0000044d3ab7ffd9
	    rip = 0x00007ff2bdd83bb4
	    Found by: given as instruction pointer in context
	 1  chrome!RawSweep [mark-compact.cc : 3404 + 0xd]
	    rbx = 0x0000000000000008   rbp = 0x0000000000000000
	    rsp = 0x00007ff2a71a88b0   r12 = 0x0000044d3ab7fff8
	    r13 = 0x0000044d3ab7fff8   r14 = 0x0000044d3ab00000
	    r15 = 0x0000044d3ab7ffd9   rip = 0x00007ff2bdd8376a
	    Found by: call frame info
	 2  chrome!ParallelSweepPage [mark-compact.cc : 3900 + 0xf]
	    rbx = 0x00000d30890c55a0   rbp = 0x0000044d3ab00000
	    rsp = 0x00007ff2a71a89b0   r12 = 0x0000000000000007
	    r13 = 0x0000044d3ab00000   r14 = 0x00000d3088b3f140
	    r15 = 0x0000000000000001   rip = 0x00007ff2bdd7cd11
	    Found by: call frame info
	 3  chrome!ParallelSweepSpace [mark-compact.cc : 3874 + 0x11]
	    rbx = 0x00000d3088b3f168   rbp = 0x00000000000007f8
	    rsp = 0x00007ff2a71a8a00   r12 = 0x0000000000000007
	    r13 = 0x0000044d3ab00000   r14 = 0x00000d3088b3f238
	    r15 = 0x00000d3088b3f258   rip = 0x00007ff2bdd7d1dd
	    Found by: call frame info
	 4  chrome!Run [mark-compact.cc : 428 + 0xb]
	    rbx = 0x0000000000000002   rbp = 0x0000000000000001
	    rsp = 0x00007ff2a71a8a60   r12 = 0x00007ff2a71a8ae8
	    r13 = 0x00007ff2c569d2b8   r14 = 0x00000d3088fdd6a0
	    r15 = 0x00007ff2a71a8ab0   rip = 0x00007ff2bdd8b2cd
	    Found by: call frame info
	 5  chrome!ThreadMain [callback.h : 68 + 0x2]
	    rbx = 0x00007ff2a71a8b40   rbp = 0x00007ff2c5648840
	    rsp = 0x00007ff2a71a8a80   r12 = 0x00007ff2a71a8ae8
	    r13 = 0x00007ff2c569d2b8   r14 = 0x00007ff2c3a4ae5f
	    r15 = 0x00007ff2a71a8ab0   rip = 0x00007ff2bebdafa8
	    Found by: call frame info
	 6  chrome!ThreadFunc [platform_thread_posix.cc : 71 + 0x8]

Knowing that some of the V8 team is in a meeting today, CC'ing mvstanton@ in case someone else can be found to investigate this.

Comment 5 by ajuma@chromium.org, Jan 27 2017

The first four frames of the stack in #4 do seem to match the original stack:
0 LiveObjectIterator::Next
1 RawSweep
2 ParallelSweepPage
3 ParallelSweepSpace
4 SweeperTask::Run

Comment 6 by kbr@chromium.org, Jan 31 2017

Cc: hpayer@chromium.org
Owner: mlippautz@chromium.org
Not to say that this isn't important to diagnose, but I see only one instance of this in the last 200 builds on this bot:

https://build.chromium.org/p/chromium.gpu.fyi/builders/Mac%2010.10%20Debug%20%28Intel%29?numbuilds=200

https://build.chromium.org/p/chromium.gpu.fyi/builders/Mac%2010.10%20Debug%20%28Intel%29/builds/16132

It does happen to be the most recent build, so it would be great if we could figure out how to diagnose this.

Comment 7 by kbr@chromium.org, Jan 31 2017

Owner: hpayer@chromium.org
Sorry, shouldn't have reassigned this bug.

Comment 8 by kbr@chromium.org, Feb 23 2017

Cc: kainino@chromium.org
Labels: -Pri-1 -Hotlist-PixelWrangler Pri-2
FYI, another instance of this bug:
https://build.chromium.org/p/chromium.gpu.fyi/builders/Win10%20Release%20%28NVIDIA%20Quadro%20M2000%29/builds/196

WebglConformance_deqp_functional_gles3_fbocolorbuffer_tex2d_01 (gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest) ... Backtrace:
	v8::internal::LiveObjectIterator<0>::Next [0x6C8FADFD+285]
	v8::internal::MarkCompactCollector::Sweeper::RawSweep [0x6C8FCB5A+970]
	v8::internal::MarkCompactCollector::Sweeper::ParallelSweepPage [0x6C8FB1D1+209]
	v8::internal::MarkCompactCollector::Sweeper::ParallelSweepSpace [0x6C8FB306+150]
	v8::internal::PagedSpace::SlowAllocateRaw [0x6C921244+244]
	v8::internal::PagedSpace::AllocateRawUnaligned [0x6C89ACC2+98]
	v8::internal::MarkCompactCollector::EvacuateVisitorBase::TryEvacuateObject [0x6C8FEC86+86]
	v8::internal::MarkCompactCollector::EvacuateOldSpaceVisitor::Visit [0x6C8FFA84+132]
	v8::internal::MarkCompactCollector::VisitLiveObjects<v8::internal::MarkCompactCollector::EvacuateOldSpaceVisitor> [0x6C8F1CD1+97]
	v8::internal::MarkCompactCollector::Evacuator::EvacuatePage [0x6C8F7D7E+510]
	v8::internal::PageParallelJob<v8::internal::EvacuationJobTraits>::Task::RunInternal [0x6C8FDD1C+60]
	base::WorkerPool::RunsTasksOnCurrentThread [0x6CDFE30E+446]
	RtlSetThreadPreferredUILanguages [0x775C2614+1460]
	EtwNotificationRegister [0x7758CD41+1857]
	BaseThreadInitThunk [0x765338F4+36]
	RtlUnicodeStringToInteger [0x775C5DE3+595]
	RtlUnicodeStringToInteger [0x775C5DAE+542]

Comment 9 by kbr@chromium.org, Apr 7 2017

Another instance:

https://build.chromium.org/p/tryserver.chromium.mac/builders/mac_optional_gpu_tests_rel/builds/8732

webgl2_conformance_tests on NVIDIA GPU on Mac Retina (with patch)
WebglConformance_deqp_functional_gles3_multisample

https://chromium-swarm.appspot.com/task?id=355ecfe024877410&refresh=10&show_raw=1

  	Thread 15 (crashed)
  	 0  Chromium Framework!__ZN2v84base2OS5AbortEv + 0x12
  	    rax = 0x0000000000000000   rdx = 0x0000000000012068
  	    rcx = 0x0000040000000503   rbx = 0x000000011dff59a3
  	    rsi = 0x000000000001cb00   rdi = 0x00007fffb0bbd048
  	    rbp = 0x0000700011f03ab0   rsp = 0x0000700011f039b8
  	     r8 = 0x0000000000000040    r9 = 0x00007fffb0bbd040
  	    r10 = 0xffffffffffffffff   r11 = 0x0000000000012068
  	    r12 = 0x00007fffb0bbda20   r13 = 0x0000700011f03b20
  	    r14 = 0x000000011dfc0398   r15 = 0x00000000000000a4
  	    rip = 0x000000011bbea472
  	    Found by: given as instruction pointer in context
  	 1  Chromium Framework!__ZN2v88internal18LiveObjectIteratorILNS0_23LiveObjectIterationModeE0EE4NextEv + 0x371
  	    rbp = 0x0000700011f03b10   rsp = 0x0000700011f03ac0
  	    rip = 0x00000001172578a1
  	    Found by: previous frame's frame pointer
  	 2  Chromium Framework!__ZN2v88internal20MarkCompactCollector7Sweeper8RawSweepEPNS0_4PageENS2_22FreeListRebuildingModeENS2_22FreeSpaceTreatmentModeE + 0x484
  	    rbp = 0x0000700011f03c00   rsp = 0x0000700011f03b20
  	    rip = 0x0000000117257034
  	    Found by: previous frame's frame pointer
  	 3  Chromium Framework!__ZN2v88internal20MarkCompactCollector7Sweeper17ParallelSweepPageEPNS0_4PageENS0_15AllocationSpaceE + 0x70
  	    rbp = 0x0000700011f03c40   rsp = 0x0000700011f03c10
  	    rip = 0x000000011724cee0
  	    Found by: previous frame's frame pointer
  	 4  Chromium Framework!__ZN2v88internal20MarkCompactCollector7Sweeper18ParallelSweepSpaceENS0_15AllocationSpaceEii + 0x1ad
  	    rbp = 0x0000700011f03ca0   rsp = 0x0000700011f03c50
  	    rip = 0x000000011724d1ed
  	    Found by: previous frame's frame pointer
  	 5  Chromium Framework!__ZN2v88internal20MarkCompactCollector7Sweeper11SweeperTask3RunEv + 0x4d
  	    rbp = 0x0000700011f03cd0   rsp = 0x0000700011f03cb0
  	    rip = 0x00000001172618cd
  	    Found by: previous frame's frame pointer
  	 6  Chromium Framework!__ZN4base12_GLOBAL__N_112WorkerThread10ThreadMainEv + 0x265
  	    rbp = 0x0000700011f03ec0   rsp = 0x0000700011f03ce0
  	    rip = 0x000000011821d285
  	    Found by: previous frame's frame pointer
  	 7  Chromium Framework!__ZN4base12_GLOBAL__N_110ThreadFuncEPv + 0x5f
  	    rbp = 0x0000700011f03ef0   rsp = 0x0000700011f03ed0
  	    rip = 0x000000011820e74f
  	    Found by: previous frame's frame pointer
...

We discovered a race in the scavenger that may result in broken free list entries, fixed in https://codereview.chromium.org/2826593004/

Ken, please notify this bug if you observe the crasher again.

Comment 11 by kbr@chromium.org, Apr 21 2017

Will do. That's great news; thanks Hannes.

Status: Fixed (was: Assigned)
This could also come from 714207. I am closing this one as well. Please re-open if you see it again.

Sign in to add a comment