New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 650314 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug



Sign in to add a comment

Fatal error in in src\v8\src\heap\mark-compact.cc, line 667: Check failed: p->SweepingDone().

Project Member Reported by kbr@chromium.org, Sep 26 2016

Issue description

Seen in this tryjob:

https://build.chromium.org/p/tryserver.chromium.win/builders/win_optional_gpu_tests_rel/builds/4017

from this CL (which did not change this test's behavior):

https://codereview.chromium.org/2297673002/

WebglConformance_deqp_functional_gles3_fragmentoutput_array_int failed:

WebglConformance_deqp_functional_gles3_fragmentoutput_array_int (gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest) ... 

#
# Fatal error in e:\b\c\b\win\src\v8\src\heap\mark-compact.cc, line 667
# Check failed: p->SweepingDone().
#
Error initializing symbols (87).  Dumping unresolved backtrace:
	61FAA1A0
	61CABE89
	61CB3F6E
	61CA2A72
	61CA2DA8
	61CD5615
	61CD31CA
	61C55034
	61C54CC5
	61C83FCE
	61C595F2
	61E4BCE9
	61E45D1C
Backtrace:
	(No symbol) [0x00000000]
	v8::base::OS::Abort [0x61FAAB2D+13]
	V8_Fatal [0x61FA820C+124]
	v8::internal::MarkCompactCollector::CollectEvacuationCandidates [0x61CABE89+313]
	v8::internal::MarkCompactCollector::StartCompaction [0x61CB3F6E+62]
	v8::internal::IncrementalMarking::StartMarking [0x61CA26A9+105]
	v8::internal::IncrementalMarking::Step [0x61CA2A72+562]
	v8::internal::IncrementalMarking::Observer::Step [0x61CA2DA8+88]
	v8::internal::NewSpace::InlineAllocationStep [0x61CD5615+101]
	v8::internal::NewSpace::EnsureAllocation [0x61CD31CA+202]
	v8::internal::NewSpace::AllocateRawUnaligned [0x61C55034+36]
	v8::internal::Heap::AllocateRaw [0x61C54CC5+437]
	v8::internal::Heap::AllocateFillerObject [0x61C83FCE+30]
	v8::internal::Factory::NewFillerObject [0x61C595F2+34]
	v8::internal::Runtime_UnwindAndFindExceptionHandler [0x61E4BCE9+13913]
	v8::internal::Runtime_AllocateInNewSpace [0x61E45D1C+204]
	(No symbol) [0x21D0625E]
	(No symbol) [0x36127A42]
	(No symbol) [0x2A9762CD]
	?end@?$HashMap@PBXV?$Eternal@VFunctionTemplate@v8@@@v8@@U?$PtrHash@$$CBX@WTF@@U?$HashTraits@PBX@4@U?$HashTraits@V?$Eternal@VFunctionTemplate@v8@@@v8@@@4@VPartitionAllocator@4@@WTF@@QAE?AU?$HashTableIteratorAdapter@V?$HashTable@PBXU?$KeyValuePair@PBXV?$Ete [0x62CEC540+26]
	v8::internal::StackGuard::ThreadLocal::Initialize [0x61C4D903+931]
	RtlFreeHeap [0x7776E023+126]
	v8::internal::Execution::Call [0x61C4D169+137]
	v8::Function::Call [0x6195DBF8+504]
	blink::V8ScriptRunner::callFunction [0x62CDE17D+421]
	blink::ScheduledAction::execute [0x63E0B17F+450]
	blink::ScheduledAction::execute [0x63E0B67F+299]
	blink::DOMTimer::fired [0x63079FDF+379]
	blink::TimerBase::runInternal [0x62C405C1+419]
	??$MakeItSo@ABQ8WebMediaPlayerMSCompositor@content@@AEXXZABV?$WeakPtr@VWebMediaPlayerMSCompositor@content@@@base@@$$V@?$InvokeHelper@$00X@internal@base@@SAXABQ8WebMediaPlayerMSCompositor@content@@AEXXZABV?$WeakPtr@VWebMediaPlayerMSCompositor@content@@@2@@ [0x61825E25+33]
	base::internal::Invoker<base::internal::BindState<void (__thiscall content::WebMediaPlayerMSCompositor::*)(void),base::WeakPtr<content::WebMediaPlayerMSCompositor> >,void __cdecl(void)>::Run [0x61826435+19]
	base::debug::TaskAnnotator::RunTask [0x623185BB+267]
	blink::scheduler::TaskQueueManager::ProcessTaskFromWorkQueue [0x62C3AE4A+594]
	blink::scheduler::TaskQueueManager::DoWork [0x62C3A47D+459]
	base::internal::FunctorTraits<void (__thiscall blink::scheduler::TaskQueueManager::*)(base::TimeTicks,bool),void>::Invoke<base::WeakPtr<blink::scheduler::TaskQueueManager> const &,base::TimeTicks const &,bool const &> [0x62C39954+34]
	base::internal::InvokeHelper<1,void>::MakeItSo<void (__thiscall blink::scheduler::TaskQueueManager::*const &)(base::TimeTicks,bool),base::WeakPtr<blink::scheduler::TaskQueueManager> const &,base::TimeTicks const &,bool const &> [0x62C3997B+37]
	base::internal::Invoker<base::internal::BindState<void (__thiscall blink::scheduler::TaskQueueManager::*)(base::TimeTicks,bool),base::WeakPtr<blink::scheduler::TaskQueueManager>,base::TimeTicks,bool>,void __cdecl(void)>::RunImpl<void (__thiscall blink::sc [0x62C39997+23]
	base::internal::Invoker<base::internal::BindState<void (__thiscall blink::scheduler::TaskQueueManager::*)(base::TimeTicks,bool),base::WeakPtr<blink::scheduler::TaskQueueManager>,base::TimeTicks,bool>,void __cdecl(void)>::Run [0x62C3B106+22]
	base::debug::TaskAnnotator::RunTask [0x623185BB+267]
	base::MessageLoop::RunTask [0x622C5FA3+1203]
	base::MessageLoop::DeferOrRunPendingTask [0x622C4C5C+60]
	base::MessageLoop::DoDelayedWork [0x622C4F94+196]
	base::MessagePumpDefault::Run [0x6231AE93+131]
	base::MessageLoop::RunHandler [0x622C5AE7+103]
	base::RunLoop::Run [0x62300879+41]
	content::RendererMain [0x637285FF+486]
	content::RunNamedProcessTypeMain [0x622A781C+176]
	content::ContentMainRunnerImpl::Run [0x622A773B+274]
	content::ContentMain [0x622A6B18+35]
	ChromeMain [0x617CA9CE+158]
	MainDllLoader::Launch [0x00EA4747+494]
	wWinMain [0x00EA2700+333]
	__scrt_common_main_seh [0x00F05927+246] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:253)
	BaseThreadInitThunk [0x75CF337A+18]
	RtlInitializeExceptionChain [0x777792B2+99]
	RtlInitializeExceptionChain [0x77779285+54]


To run the tests (not sure how frequently this reproduces: build Release with dcheck_always_on=true, then:

./content/test/gpu/run_gpu_integration_test.py webgl_conformance --browser=release --webgl-conformance-version=2.0.0

 
Cc: epertoso@chromium.org eisinger@chromium.org
Status: Available (was: Untriaged)

Comment 3 by kbr@chromium.org, Oct 25 2016

Labels: -Pri-2 Pri-1
Increasing to P1. This looks like a recent regression.

Comment 4 by jochen@chromium.org, Oct 26 2016

Cc: -eisinger@chromium.org jochen@chromium.org
Owner: mlippautz@chromium.org
Status: Assigned (was: Available)
Labels: Performance-Memory
Owner: u...@chromium.org
Assigning to current sheriff.

Comment 6 by u...@chromium.org, Oct 26 2016

We changed how sweeping starts and ends in https://codereview.chromium.org/2445283003/ yesterday.

I'll keep an eye on the buildbot to see if the assert reproduces with the new code.
Still happens
https://build.chromium.org/p/chromium.gpu.fyi/builders/Win8%20Release%20%28NVIDIA%29/builds/24999

WebglConformance_deqp_functional_gles3_texturespecification_basic_teximage2d_2d_00 (gpu_tests.webgl_conformance_integration_test.WebGLConformanceIntegrationTest) ... 

#
# Fatal error in e:\b\c\b\win\src\v8\src\heap\mark-compact.cc, line 649
# Check failed: p->SweepingDone().
#
Error initializing symbols (87).  Dumping unresolved backtrace:
	6B1310F0
	6A493F2B
	6A49C76E
	6A48A9CA
	6A487DA6
	6A486F03
	6A9C1878
	6B26028F
	6B25F7C6
	6B25EA43
	6B25EA89
	6B25EAA5
	6B260568
	6A9C1878
	6A986DDC
	6A985D65
	6A9C41C0
	6A986901
	6A996679
	6BD84838
	6A93A9A4
	6A93A8C3
	6A939D44
	69F994EE
	010D4C9E
	010D2954
	0111CF58
	77387C04
	777FAD1F
	777FACEA
Backtrace:
	(No symbol) [0x00000000]
	v8::base::OS::Abort [0x6B12992D+13]
	V8_Fatal [0x6B12944C+124]
	v8::internal::MarkCompactCollector::CollectEvacuationCandidates [0x6A493F2B+315]
	v8::internal::MarkCompactCollector::StartCompaction [0x6A49C76E+62]
	v8::internal::IncrementalMarking::StartMarking [0x6A48A669+105]
	v8::internal::IncrementalMarking::Step [0x6A48A9CA+490]
	v8::internal::IncrementalMarking::AdvanceIncrementalMarking [0x6A487DA6+150]
	v8::internal::IncrementalMarkingJob::Task::RunInternal [0x6A486F03+195]
	base::debug::TaskAnnotator::RunTask [0x6A9C1878+280]
	blink::scheduler::TaskQueueManager::ProcessTaskFromWorkQueue [0x6B26028F+715]
	blink::scheduler::TaskQueueManager::DoWork [0x6B25F7C6+462]
	base::internal::FunctorTraits<void (__thiscall blink::scheduler::TaskQueueManager::*)(base::TimeTicks,bool),void>::Invoke<base::WeakPtr<blink::scheduler::TaskQueueManager> const &,base::TimeTicks const &,bool const &> [0x6B25EA43+34]
	base::internal::InvokeHelper<1,void>::MakeItSo<void (__thiscall blink::scheduler::TaskQueueManager::*const &)(base::TimeTicks,bool),base::WeakPtr<blink::scheduler::TaskQueueManager> const &,base::TimeTicks const &,bool const &> [0x6B25EA89+37]
	base::internal::Invoker<base::internal::BindState<void (__thiscall blink::scheduler::TaskQueueManager::*)(base::TimeTicks,bool),base::WeakPtr<blink::scheduler::TaskQueueManager>,base::TimeTicks,bool>,void __cdecl(void)>::RunImpl<void (__thiscall blink::sc [0x6B25EAA5+23]
	base::internal::Invoker<base::internal::BindState<void (__thiscall blink::scheduler::TaskQueueManager::*)(base::TimeTicks,bool),base::WeakPtr<blink::scheduler::TaskQueueManager>,base::TimeTicks,bool>,void __cdecl(void)>::Run [0x6B260568+22]
	base::debug::TaskAnnotator::RunTask [0x6A9C1878+280]
	base::MessageLoop::RunTask [0x6A986DDC+1228]
	base::MessageLoop::DoWork [0x6A985D65+597]
	base::MessagePumpDefault::Run [0x6A9C41C0+416]
	base::MessageLoop::RunHandler [0x6A986901+305]
	base::RunLoop::Run [0x6A996679+41]
	content::RendererMain [0x6BD84838+486]
	content::RunNamedProcessTypeMain [0x6A93A9A4+176]
	content::ContentMainRunnerImpl::Run [0x6A93A8C3+274]
	content::ContentMain [0x6A939D44+35]
	ChromeMain [0x69F994EE+158]
	MainDllLoader::Launch [0x010D4C9E+527]
	wWinMain [0x010D2954+342]
	__scrt_common_main_seh [0x0111CF58+246] (f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl:253)
	BaseThreadInitThunk [0x77387C04+36]
	RtlInitializeExceptionChain [0x777FAD1F+143]
	RtlInitializeExceptionChain [0x777FACEA+90]

Comment 9 by u...@chromium.org, Oct 28 2016

Cannot reproduce locally.

Doing trybot debugging with more strick checks in: 

https://codereview.chromium.org/2462613002/
Project Member

Comment 10 by bugdroid1@chromium.org, Oct 28 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/b0b1d7a9e6e199978c2c92aebcc592e720743de2

commit b0b1d7a9e6e199978c2c92aebcc592e720743de2
Author: ulan <ulan@chromium.org>
Date: Fri Oct 28 16:53:13 2016

[heap] Check that sweeping is completed when collecting evacuation
candidates.

BUG= chromium:650314 

Review-Url: https://codereview.chromium.org/2459003002
Cr-Commit-Position: refs/heads/master@{#40651}

[modify] https://crrev.com/b0b1d7a9e6e199978c2c92aebcc592e720743de2/src/heap/mark-compact.cc

Comment 12 by u...@chromium.org, Nov 8 2016

Thanks, ynovikov@!

New DCHECKs not firing eliminates one hypothesis. Investigation continues.

Comment 13 by u...@chromium.org, Nov 8 2016

Is the bot in #11 running with dchecks on?

Comment 14 by kbr@chromium.org, Nov 8 2016

#13: Yes, it is. Look at a recent build from:
https://build.chromium.org/p/chromium.gpu.fyi/builders/Mac%2010.10%20Release%20%28Intel%29/

the parent builder is "GPU Mac Builder" (just noticed this is stuck, filed Issue 663464 about it):
https://build.chromium.org/p/chromium.gpu.fyi/builders/GPU%20Mac%20Builder

Look at the stdio for "generate build files" from any build:
https://build.chromium.org/p/chromium.gpu.fyi/builders/GPU%20Mac%20Builder/builds/75864/steps/generate_build_files/logs/stdio

Note that:
dcheck_always_on = true

is set in the GN configuration.

Issue 663660 has been merged into this issue.
Project Member

Comment 16 by sheriffbot@chromium.org, Nov 9 2016

Labels: Fracas


If this update was incorrect, please add "Fracas-Wrong" label to prevent future updates.

- Go/Fracas

Comment 17 by u...@chromium.org, Nov 9 2016

Thanks, Ken. Good to know how to check the flags in future :)

I found one scenario that can lead to this assert:
https://codereview.chromium.org/2484153004/
Project Member

Comment 18 by bugdroid1@chromium.org, Nov 9 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/b621987195cdb1b2031809d9f10aff3005190161

commit b621987195cdb1b2031809d9f10aff3005190161
Author: ulan <ulan@chromium.org>
Date: Wed Nov 09 12:14:15 2016

[heap] Ensure that the sweeper does not lose unswept pages.

This fixes a race between the sweeper and the array buffer tracker
that causes the sweeper to skip unswept pages.

The scenario:
1. Mark-compact GC adds page p to the sweeping_list_ of the sweeper.
2. GC finishes, the main thread starts executinng JS.
3. The main thread takes p->mutex to unregister an array buffer.
4. A sweeper thread removes p from the sweeping_list_ and tries to
   take p->mutex. The try fails. The sweeper drops p and continues
   to the next page.
5. During selection of evacuation candidate in the next GC we hit
   page->SweepingDone() assert.

BUG= chromium:650314 

Review-Url: https://codereview.chromium.org/2484153004
Cr-Commit-Position: refs/heads/master@{#40857}

[modify] https://crrev.com/b621987195cdb1b2031809d9f10aff3005190161/src/heap/mark-compact.cc

Comment 19 by kbr@chromium.org, Nov 9 2016

Ulan, this is just awesome! Thank you for tracking down this nasty race condition!

Comment 20 by u...@chromium.org, Nov 11 2016

Status: Fixed (was: Assigned)

Sign in to add a comment