New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 783157 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Nov 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 1
Type: Bug

Blocking:
issue 781585



Sign in to add a comment

Windows WebGL tests flakily failing with an assert in scheduler.cc

Project Member Reported by jmad...@chromium.org, Nov 9 2017

Issue description

Breaking many bots, see:

https://ci.chromium.org/buildbot/chromium.gpu.fyi/Win7%20Release%20%28NVIDIA%29/
https://ci.chromium.org/buildbot/chromium.gpu.fyi/Win7%20Release%20%28AMD%29/
https://ci.chromium.org/buildbot/chromium.gpu.fyi/Win7%20Release%20%28NVIDIA%20Quadro%20P400%29/

First failing builds:

https://ci.chromium.org/buildbot/chromium.gpu.fyi/Win7%20Release%20%28NVIDIA%29/32046
https://ci.chromium.org/buildbot/chromium.gpu.fyi/Win7%20Release%20%28AMD%29/5493
https://ci.chromium.org/buildbot/chromium.gpu.fyi/Win7%20Release%20%28NVIDIA%20Quadro%20P400%29/218

However since this is a flaky failure I'm actually suspecting
"gpu scheduler: Compute stream priorities based on priority of waiters."

https://chromium-review.googlesource.com/754415

From
https://ci.chromium.org/buildbot/chromium.gpu.fyi/Win7%20Release%20%28NVIDIA%29/32045
https://ci.chromium.org/buildbot/chromium.gpu.fyi/Win7%20Release%20%28AMD%29/5493

Error text:
[1840:4036:1109/013109.655:FATAL:scheduler.cc(366)] Check failed: waiting_priority_counts_[static_cast<int>(priority)] > 0. 
Backtrace:
	base::debug::StackTrace::StackTrace [0x66413A20+32]
	base::debug::StackTrace::StackTrace [0x6640AB7D+13]
	logging::LogMessage::~LogMessage [0x6638FCEE+78]
	gpu::Scheduler::Sequence::RemoveWaitingPriority [0x67F6616F+119]
	gpu::Scheduler::Sequence::RemoveWaitFence [0x67F67149+85]
	gpu::Scheduler::SyncTokenFenceReleased [0x67F67E4E+58]
	base::internal::Invoker<base::internal::BindState<void (__thiscall gpu::Scheduler::*)(gpu::SyncToken const &,unsigned int,gpu::IdType<gpu::SyncPointOrderData,unsigned int,0>,gpu::IdType<gpu::SyncPointOrderData,unsigned int,0>),base::WeakPtr<gpu::Scheduler [0x67F69773+59]
	gpu::SyncPointClientState::ReleaseFenceSyncHelper [0x67F1EA8F+263]
	gpu::SyncPointClientState::ReleaseFenceSync [0x67F1F21E+112]
	gpu::GpuCommandBufferStub::OnFenceSyncRelease [0x680524E7+125]
	gpu::gles2::GLES2DecoderPassthroughImpl::DoInsertFenceSyncCHROMIUM [0x67FBAE0D+17]
	gpu::gles2::GLES2DecoderPassthroughImpl::HandleInsertFenceSyncCHROMIUM [0x67FAEC42+32]
	gpu::gles2::GLES2DecoderPassthroughImpl::DoCommandsImpl<0> [0x67F7C540+192]
	gpu::gles2::GLES2DecoderPassthroughImpl::DoCommands [0x67F7C16F+33]
	gpu::CommandBufferService::Flush [0x67F285F1+353]
	gpu::GpuCommandBufferStub::OnAsyncFlush [0x6804F553+307]
	IPC::MessageT<GpuCommandBufferMsg_AsyncFlush_Meta,std::tuple<int,unsigned int,std::vector<ui::LatencyInfo,std::allocator<ui::LatencyInfo> > >,void>::Dispatch<gpu::GpuCommandBufferStub,gpu::GpuCommandBufferStub,void,void (__thiscall gpu::GpuCommandBufferSt [0x6804F36E+102]
	gpu::GpuCommandBufferStub::OnMessageReceived [0x6804E452+1234]
	gpu::GpuChannel::HandleMessageHelper [0x6804C931+45]
<snip>

This is affecting the ability to land ANGLE CLs so marking as pri-1. Assigning to CL author (Victor). We might need to revert this CL or make a quick fix.
 
Blocking: 781585
Going to try reverting that CL.
Project Member

Comment 2 by bugdroid1@chromium.org, Nov 9 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4a3cd6261a6396e364791c5dd9b043f755f84785

commit 4a3cd6261a6396e364791c5dd9b043f755f84785
Author: Jamie Madill <jmadill@chromium.org>
Date: Thu Nov 09 14:10:13 2017

Revert "gpu scheduler: Compute stream priorities based on priority of waiters."

This reverts commit 56eadad0e85ad5331dea863690efc604e3d763b4.

Reason for revert: Seems to be breaking the Windows WebGL tests flakily. Possible race condition.

[1840:4036:1109/013109.655:FATAL:scheduler.cc(366)] Check failed: waiting_priority_counts_[static_cast<int>(priority)] > 0. 
Backtrace:
	base::debug::StackTrace::StackTrace [0x66413A20+32]
	base::debug::StackTrace::StackTrace [0x6640AB7D+13]
	logging::LogMessage::~LogMessage [0x6638FCEE+78]
	gpu::Scheduler::Sequence::RemoveWaitingPriority [0x67F6616F+119]
	gpu::Scheduler::Sequence::RemoveWaitFence [0x67F67149+85]
	gpu::Scheduler::SyncTokenFenceReleased [0x67F67E4E+58]
	base::internal::Invoker<base::internal::BindState<void (__thiscall gpu::Scheduler::*)(gpu::SyncToken const &,unsigned int,gpu::IdType<gpu::SyncPointOrderData,unsigned int,0>,gpu::IdType<gpu::SyncPointOrderData,unsigned int,0>),base::WeakPtr<gpu::Scheduler [0x67F69773+59]
	gpu::SyncPointClientState::ReleaseFenceSyncHelper [0x67F1EA8F+263]
	gpu::SyncPointClientState::ReleaseFenceSync [0x67F1F21E+112]
	gpu::GpuCommandBufferStub::OnFenceSyncRelease [0x680524E7+125]
	gpu::gles2::GLES2DecoderPassthroughImpl::DoInsertFenceSyncCHROMIUM [0x67FBAE0D+17]
	gpu::gles2::GLES2DecoderPassthroughImpl::HandleInsertFenceSyncCHROMIUM [0x67FAEC42+32]
	gpu::gles2::GLES2DecoderPassthroughImpl::DoCommandsImpl<0> [0x67F7C540+192]
	gpu::gles2::GLES2DecoderPassthroughImpl::DoCommands [0x67F7C16F+33]
<snip>

Example build:
https://ci.chromium.org/buildbot/chromium.gpu.fyi/Win7%20Release%20%28AMD%29/5493

Also affects other Windows configurations.

BUG= 781585 , 783157 

Original change's description:
> gpu scheduler: Compute stream priorities based on priority of waiters.
> 
> Previously, a stream tracked all of it's fences that had waiting streams
> in 'release_fences'.  If 'release_fences' was non-empty, the stream's
> priority got bumped up to High.  This means a stream with Normal priority
> waiting on Low priority stream could bump that stream up to High priority.
> 
> This change removes that tracking, and instead keeps count of the number
> of waiting streams at each StreamPriority.  The highest priority with non-
> zero count becomes the stream's priority.
> 
> When a stream changes priority, it recursively propagates it's new priority
> to all streams it's waiting on.
> 
> BUG= 781585 
> 
> Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
> Change-Id: I1fc96f6ca065a8e1f83ddaa61ae4f9725c324bcb
> Reviewed-on: https://chromium-review.googlesource.com/754415
> Commit-Queue: Victor Miura <vmiura@chromium.org>
> Reviewed-by: Sunny Sachanandani <sunnyps@chromium.org>
> Reviewed-by: Antoine Labour <piman@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#515097}

TBR=vmiura@chromium.org,sunnyps@chromium.org,piman@chromium.org

Change-Id: I398a59a1c039030f0ba0b915d6cb1a1dc582158a
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug:  781585 
Cq-Include-Trybots: master.tryserver.chromium.android:android_optional_gpu_tests_rel;master.tryserver.chromium.linux:linux_optional_gpu_tests_rel;master.tryserver.chromium.mac:mac_optional_gpu_tests_rel;master.tryserver.chromium.win:win_optional_gpu_tests_rel
Reviewed-on: https://chromium-review.googlesource.com/760456
Reviewed-by: Jamie Madill <jmadill@chromium.org>
Commit-Queue: Jamie Madill <jmadill@chromium.org>
Cr-Commit-Position: refs/heads/master@{#515164}
[modify] https://crrev.com/4a3cd6261a6396e364791c5dd9b043f755f84785/gpu/command_buffer/service/scheduler.cc
[modify] https://crrev.com/4a3cd6261a6396e364791c5dd9b043f755f84785/gpu/command_buffer/service/scheduler_unittest.cc

Status: Fixed (was: Assigned)
The revert seems to have cleared up the bot failures.

Sign in to add a comment