Crash is in a vector owned by CSS. Over to the CSS team to take a look.

We don't really have the resources to look into this at the moment I'm afraid.

Jamie, anecdotally, how often is this happening?

Unfortunately there doesn't seem to be data on chromium-try-flakes for this test suite:

https://chromium-try-flakes.appspot.com/search?q=webgl2_conformance_gl_tests

CC'ing stgao@ - what would we need to do to get this to show up there? I'm not aware of any disabling of flakiness dashboard uploads that we consciously did on https://ci.chromium.org/p/chromium/g/chromium.gpu.fyi/console . Thanks.

I haven't seen any other flakes with this check failing. Agree it would be great to make this test suite show up on chromium-try-flakes, it has problems with the flakiness not being as visible.

Thanks Jamie for your feedback. In that case I agree that we can leave this as P2 or even P3.

stgao@ how do we enable flakiness dashboard uploads for the chromium.gpu.fyi bots and ANGLE tryservers?

I will look into why chromium-try-flakes can't detect this flake.

Regarding "flakiness dashboard uploads", I assume that it is to upload test results to test-results.appspot.com app. This is a prerequisite for chromium-try-flakes to support flake detection for CQ runs.
But seanmccullough@ is a better person for this question.

Aha. So the reason the tests don't show us is the test results upload step fails because of the old issue 568149. 

chromium-try-flakes currently only scans CQ builds in the project "chromium/chromium/src".

I assume that tryserver.chromium.angle is in another project. What's the project name of those CQ jobs, "angle/angle"?
On CTF side, I need to whitelist the project name at least, but some substantial change might be needed for data model as well.
I will dig further once test results are uploaded.

+wylieb@, the angle project is another user for flake detection. Will the BQ-based new flake detection be able to support it smoothly?

Shouldn't be a problem as long as the results from angle tests adhere to PASS/FAIL standards. If it's different in a meaningful way (like webkit layout tests), then there might have to be special rules applied. 

Either way, it's doable. 

I filed a separate bug 828043 to support flake detection for ANGLE project.

One particular question is: are test results from ANGLE uploaded to test-results.appspot.com ?

An update! I have seen a couple more crashes with a similar check failing:

This is in webgl_conformance_d3d9_tests on ATI GPU in the text WebglConformance_conformance_textures_misc_texture_upload_size.

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win7%20FYI%20Debug%20%28AMD%29/163
https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Win7%20FYI%20Debug%20%28AMD%29/177

https://chromium-swarm.appspot.com/task?id=3cec9efc06837f10&refresh=10&show_raw=1
https://chromium-swarm.appspot.com/task?id=3cf7359768583a10&refresh=10&show_raw=1

Both have this pattern:

[3936:4976:0419/080841.886:FATAL:heap_page.h(1066)] Check failed: IsMarked(). 
Backtrace:
	base::debug::StackTrace::StackTrace [0x6E2C4496+102]
	base::debug::StackTrace::StackTrace [0x6E2C318B+27]
	logging::LogMessage::~LogMessage [0x6E338D54+148]
	blink::HeapObjectHeader::Unmark [0x04D4D6D1+193]
	blink::NormalPageArena::PromptlyFreeObjectInFreeList [0x055DB5DA+186]
	blink::NormalPageArena::ShrinkObject [0x055DC051+1153]
	blink::HeapAllocator::BackingShrink [0x055C4C07+983]
	blink::HeapAllocator::ShrinkVectorBacking [0x055C4DB4+52]
	WTF::VectorBuffer<blink::RuleData,0,blink::HeapAllocator>::ShrinkBuffer [0x076D469A+314]
	WTF::Vector<blink::RuleData,0,blink::HeapAllocator>::ShrinkCapacity [0x076D4285+133]
	WTF::Vector<blink::RuleData,0,blink::HeapAllocator>::ShrinkToFit [0x076C4A8F+31]
	blink::RuleSet::CompactRules [0x076C0EDC+364]
	blink::RuleSet::CompactRulesIfNeeded [0x06DD9C92+50]
	blink::MatchRequest::MatchRequest [0x0764472A+122]
	blink::StyleResolver::MatchRuleSet [0x07681DDC+92]
	blink::StyleResolver::MatchUARules [0x07681C2D+125]
	blink::StyleResolver::MatchAllRules [0x07681ECC+60]
	blink::StyleResolver::StyleForElement [0x0768319C+2140]
	blink::SVGElement::CustomStyleForLayoutObject [0x0916ADAB+107]
	blink::Element::StyleForLayoutObject [0x078C8537+311]
	blink::Element::RecalcOwnStyle [0x078C9669+841]
	blink::Element::RecalcStyle [0x078C8F6B+1019]
	blink::Document::UpdateStyle [0x077E9EC9+1609]
	blink::Document::UpdateStyleAndLayoutTree [0x077E4FAD+1085]
	blink::Document::FinishedParsing [0x0780561F+767]
	blink::XMLDocumentParser::end [0x0935FB0F+671]
	blink::XMLDocumentParser::Finish [0x093600CB+91]
	blink::LocalFrame::ForceSynchronousDocumentInstall [0x07FBD9A9+713]
	blink::SVGImage::DataChanged [0x0911AA2F+3151]
	blink::Image::SetData [0x05343960+224]
	blink::ImageResourceContent::UpdateImage [0x08CBB7E7+1495]
	blink::ImageResource::UpdateImage [0x08CB0362+194]
	blink::ImageResource::Finish [0x08CB1A75+213]
	blink::ResourceFetcher::HandleLoaderFinish [0x0567C7E8+2184]

<snip stack>

So this is probably a real bug affecting users at times. Note that this test is already marked as flaky in several other configurations, so it probably has a broader reach than just AMD D3D9.

Another incidence:

https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20FYI%20Release%20%28Intel%20HD%20630%29/5770

WebglConformance_conformance2_textures_svg_image_tex_2d_r16f_red_half_float failed in webgl2_conformance_tests on Intel GPU on Linux on Ubuntu:

https://chromium-swarm.appspot.com/task?id=3ff18a4edcd7f810&refresh=10&show_raw=1

[1:1:0914/083552.215597:FATAL:heap_page.h(989)] Check failed: IsMarked(). 
#0 0x556914e5a3ac base::debug::StackTrace::StackTrace()
#1 0x556914da83fb logging::LogMessage::~LogMessage()
#2 0x55691475b089 blink::HeapObjectHeader::Unmark()
#3 0x55691475ae54 blink::NormalPageArena::PromptlyFreeObjectInFreeList()
#4 0x55691475b7b6 blink::NormalPageArena::ShrinkObject()
#5 0x556914752107 blink::HeapAllocator::BackingShrink()
#6 0x55691810ea8d WTF::Vector<>::ShrinkCapacity()
#7 0x5569181092ed blink::RuleSet::CompactRules()
#8 0x556917f46d7d blink::MatchRequest::MatchRequest()
#9 0x55691807c22e blink::StyleResolver::MatchUARules()
#10 0x55691807c345 blink::StyleResolver::MatchAllRules()
#11 0x55691807db43 blink::StyleResolver::StyleForElement()
#12 0x556919050c64 blink::SVGElement::CustomStyleForLayoutObject()
#13 0x556918030e6b blink::Element::StyleForLayoutObject()
#14 0x556918031c2f blink::Element::RecalcOwnStyle()
#15 0x5569180314ed blink::Element::RecalcStyle()
#16 0x556917f47446 blink::StyleEngine::RecalcStyle()
#17 0x556917ef95d9 blink::Document::UpdateStyle()
#18 0x556917ef437b blink::Document::UpdateStyleAndLayoutTree()
#19 0x556917f0c37b blink::Document::FinishedParsing()
#20 0x55691913d891 blink::XMLDocumentParser::end()
#21 0x5569187b678f blink::LocalFrame::ForceSynchronousDocumentInstall()
#22 0x55691901a86d blink::SVGImage::DataChanged()
#23 0x556918e14926 blink::ImageResourceContent::UpdateImage()
#24 0x556918e113f3 blink::ImageResource::Finish()
#25 0x5569147a2223 blink::ResourceFetcher::HandleLoaderFinish()
#26 0x5569147bbbad blink::ResourceLoader::DidFinishLoading()
#27 0x556919804b4d content::WebURLLoaderImpl::Context::OnCompletedRequest()
#28 0x5569198054b8 content::WebURLLoaderImpl::RequestPeerImpl::OnCompletedRequest()
#29 0x55691941112d content::ResourceDispatcher::OnRequestComplete()
#30 0x55691280a4e1 content::ThrottlingURLLoader::OnComplete()
#31 0x5569121facaa network::mojom::URLLoaderClientStubDispatch::Accept()
#32 0x556914ec49d2 mojo::InterfaceEndpointClient::HandleValidatedMessage()
#33 0x556914ec71f6 mojo::FilterChain::Accept()
#34 0x556914ec5ed2 mojo::InterfaceEndpointClient::HandleIncomingMessage()
#35 0x556914ecd61c mojo::internal::MultiplexRouter::ProcessIncomingMessage()
#36 0x556914ecc9e0 mojo::internal::MultiplexRouter::Accept()
#37 0x556914ec71f6 mojo::FilterChain::Accept()
#38 0x556914ec240e mojo::Connector::ReadSingleMessage()
#39 0x556914ec3034 mojo::Connector::ReadAllAvailableMessages()
#40 0x556914ec2e96 mojo::Connector::OnHandleReadyInternal()
#41 0x5569127ecc44 mojo::SimpleWatcher::DiscardReadyState()
#42 0x556914ee44d3 mojo::SimpleWatcher::OnHandleReady()
#43 0x5569129de53c _ZN4base8internal7InvokerINS0_9BindStateIMN3viz14GpuServiceImplEFvN3gfx21GenericSharedMemoryIdEiRKN3gpu9SyncTokenEEJNS_7WeakPtrIS4_EES6_iS8_EEEFvvEE7RunImplISC_NSt3__15tupleIJSE_S6_iS8_EEEJLm0ELm1ELm2ELm3EEEEvOT_OT0_NSJ_16integer_sequenceImJXspT1_EEEE
#44 0x556914db09bd base::debug::TaskAnnotator::RunTask()
#45 0x556914e046d0 base::sequence_manager::internal::ThreadControllerImpl::DoWork()
#46 0x5569120f9138 _ZN4base8internal7InvokerINS0_9BindStateIMN3net14MDnsConnectionEFviEJNS_7WeakPtrIS4_EEiEEEFvvEE3RunEPNS0_13BindStateBaseE
#47 0x556914db09bd base::debug::TaskAnnotator::RunTask()
#48 0x556914dafa7e base::MessageLoop::RunTask()
#49 0x556914dafe82 base::MessageLoop::DoWork()
#50 0x556914db35c6 base::MessagePumpDefault::Run()
#51 0x556914daf551 base::MessageLoop::Run()
#52 0x556914dd9ae6 base::RunLoop::Run()
#53 0x556919f119d2 content::RendererMain()
#54 0x556914977a87 content::RunZygote()
#55 0x556914978c97 content::ContentMainRunnerImpl::Run()
#56 0x5569149add59 service_manager::Main()
#57 0x556914977041 content::ContentMain()
#58 0x556911d541b3 ChromeMain
#59 0x7fec731903f1 __libc_start_main
#60 0x556911d5402a _start

Linking to related Issue 846967.

This looks like a real bug. Will have a detailed look on Monday.

blink::RuleSet is the only user of (Heap)TerminatedArray and we are aware that there seems to be a problem with that custom data structure (array where each element stores whether it is the last one...).

Keishi is/was working on migrating that to HeapVector. What's the progress there?

I immediately see that we do shrink a backing store through a different path than the regular one (which is PromptlyFreeObject). 

At first sight I do not see why that object would need to be marked. The path through BackingShrink tries to promptly free the last part of the vector into the free list. This object is actually never marked as it was just created the call before in ShrinkObject.

I see that there are two paths into PromptlyFreeObjectInFreeList:
1. PromptlyFreeObject
2. ShrinkObject

The path in (1) deals with an object that could be marked.

The path in (2) deals with a freshly created object in [1]. This object is actually never marked.

So the code in PromptlyFreeObjectInFreeListshould actually be:
  if (header->IsMarked()) header->Unmark()

Will try to create a test to double check and fix on Monday.

[1] https://cs.chromium.org/chromium/src/third_party/blink/renderer/platform/heap/heap_page.cc?type=cs&g=0&l=810

> So the code in PromptlyFreeObjectInFreeListshould actually be:
>  if (header->IsMarked()) header->Unmark()

Looks like a reasonable change :)

Issue 846967 has been merged into this issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/af96b5268a067e00808bc75701e2381f2fc2bb7d

commit af96b5268a067e00808bc75701e2381f2fc2bb7d
Author: Michael Lippautz <mlippautz@chromium.org>
Date: Mon Sep 17 16:03:47 2018

[heap] Fix promptly freeing during object shrinking

Fixes an issue where the object was assumed to be marked during promptly
freeing on a page that has not been swept yet. There's an alternative
path into this method though for which a new object header has been
created by carving off unusued memory in which case the header is not
marked.

Bug:  823289 
Change-Id: Ia0dc203e285ccf297cca582aae35caacb74dde1e
Reviewed-on: https://chromium-review.googlesource.com/1227940
Reviewed-by: Kentaro Hara <haraken@chromium.org>
Commit-Queue: Michael Lippautz <mlippautz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#591693}
[modify] https://crrev.com/af96b5268a067e00808bc75701e2381f2fc2bb7d/third_party/blink/renderer/platform/heap/heap_page.cc
[modify] https://crrev.com/af96b5268a067e00808bc75701e2381f2fc2bb7d/third_party/blink/renderer/platform/heap/heap_test.cc
[modify] https://crrev.com/af96b5268a067e00808bc75701e2381f2fc2bb7d/third_party/blink/renderer/platform/heap/heap_test_utilities.cc
[modify] https://crrev.com/af96b5268a067e00808bc75701e2381f2fc2bb7d/third_party/blink/renderer/platform/heap/heap_test_utilities.h

Crash should only happen on DCHECK builds, so I don't think there's anything to backmerge.

Thank you Michael for getting to the bottom of this!