Failure on Mac GPU WebGL CTS: WebglConformance_deqp_functional_gles3_vertexarrays_single_attribute_output_type_short |
||||||||
Issue descriptionThis started failing here: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28Intel%29/327 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Release%20%28Intel%29/6072 Stack trace is a bit hard to parse but seems to crashing in "__ZN13GrCCPathCache5evictEP18GrCCPathCacheEntry" which seems to deal with Skia's path caching. This CL seems related: cc: Cache text blobs in the GPU service. by Khushal https://chromium-review.googlesource.com/c/chromium/src/+/1292722 The Skia roll didn't seem to have anything suspicious. Will try a revert.
,
Oct 21
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e99280223d68ac7886976c45c562ce4dcbd58d54 commit e99280223d68ac7886976c45c562ce4dcbd58d54 Author: Jamie Madill <jmadill@chromium.org> Date: Sun Oct 21 15:26:29 2018 Revert "cc: Cache text blobs in the GPU service." This reverts commit 57df06767601e98a1b51239d3a5f3862e3d000dd. Reason for revert: Seems to be causing a crash on Mac. https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28Intel%29/327 https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Release%20%28Intel%29/6072 Crash in __ZN13GrCCPathCache5evictEP18GrCCPathCacheEntry Bug: 897507 Original change's description: > cc: Cache text blobs in the GPU service. > > Since skia's internal caching is keyed on text blobs, its important to > reuse the same object. Make sure we cache text blobs in the > ServiceTransferCache to get a cache hit in skia. > > R=enne@chromium.org > > Bug: 894200 > Change-Id: I8808d649ca3520211d034e07e7d4670ffb39094c > Reviewed-on: https://chromium-review.googlesource.com/c/1292722 > Commit-Queue: Khushal <khushalsagar@chromium.org> > Reviewed-by: enne <enne@chromium.org> > Cr-Commit-Position: refs/heads/master@{#601404} TBR=enne@chromium.org,khushalsagar@chromium.org # Not skipping CQ checks because original CL landed > 1 day ago. Bug: 894200 Change-Id: I5e0a151c5a539aa8a9b6cac5f9bcf710ba2f405a Reviewed-on: https://chromium-review.googlesource.com/c/1292384 Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Jamie Madill <jmadill@chromium.org> Cr-Commit-Position: refs/heads/master@{#601451} [modify] https://crrev.com/e99280223d68ac7886976c45c562ce4dcbd58d54/cc/paint/BUILD.gn [modify] https://crrev.com/e99280223d68ac7886976c45c562ce4dcbd58d54/cc/paint/paint_op_reader.cc [modify] https://crrev.com/e99280223d68ac7886976c45c562ce4dcbd58d54/cc/paint/paint_op_writer.cc [delete] https://crrev.com/d2a371815c3881a02c05c2de63f33daf1f9bd4af/cc/paint/textblob_transfer_cache_entry.cc [delete] https://crrev.com/d2a371815c3881a02c05c2de63f33daf1f9bd4af/cc/paint/textblob_transfer_cache_entry.h [modify] https://crrev.com/e99280223d68ac7886976c45c562ce4dcbd58d54/cc/paint/transfer_cache_entry.cc [modify] https://crrev.com/e99280223d68ac7886976c45c562ce4dcbd58d54/cc/paint/transfer_cache_entry.h
,
Oct 21
Seems to have fixed the failure.
,
Oct 22
This looks like a skia internal LRU cache bug in GrCCPathCache. [8895:775:1020/013503.464530:INFO:GrCCPathCache.cpp(191)] ../../third_party/skia/src/gpu/ccpr/GrCCPathCache.cpp:191: fatal error: "assert(isInCache == fLRU.isInList(entry))" Received signal 6 0 Chromium Framework 0x000000011780cd6f base::debug::StackTrace::StackTrace(unsigned long) + 31 1 Chromium Framework 0x000000011780cbc1 base::debug::(anonymous namespace)::StackDumpSignalHandler(int, __siginfo*, void*) + 2385 2 libsystem_platform.dylib 0x00007fff74209b3d _sigtramp + 29 3 ??? 0x0000000000000000 0x0 + 0 4 libsystem_c.dylib 0x00007fff740c81c9 abort + 127 5 Chromium Framework 0x0000000117c324b5 sk_abort_no_print() + 21 6 Chromium Framework 0x0000000117e15c02 GrCCPathCache::evict(GrCCPathCacheEntry*) + 98 7 Chromium Framework 0x0000000117e1623b GrCCPathCache::purgeAsNeeded() + 91 8 Chromium Framework 0x0000000117e26eda GrCoverageCountingPathRenderer::postFlush(GrDeferredUploadToken, unsigned int const*, int) + 570 9 Chromium Framework 0x0000000117c9dd3c GrDrawingManager::flush(GrSurfaceProxy*, int, GrBackendSemaphore*) + 2956 10 Chromium Framework 0x0000000117c9e719 GrDrawingManager::prepareSurfaceForExternalIO(GrSurfaceProxy*, int, GrBackendSemaphore*) + 185 11 Chromium Framework 0x0000000117cd2382 GrRenderTargetContext::prepareForExternalIO(int, GrBackendSemaphore*) + 290 12 Chromium Framework 0x0000000117def2ea SkGpuDevice::flushAndSignalSemaphores(int, GrBackendSemaphore*) + 58 13 Chromium Framework 0x0000000119445b60 gpu::raster::RasterDecoderImpl::DoEndRasterCHROMIUM() + 320 14 Chromium Framework 0x0000000119438369 gpu::raster::RasterDecoderImpl::HandleEndRasterCHROMIUM(unsigned int, void const volatile*) + 9 15 Chromium Framework 0x000000011943dc70 gpu::error::Error gpu::raster::RasterDecoderImpl::DoCommandsImpl<false>(unsigned int, void const volatile*, int, int*) + 512 16 Chromium Framework 0x0000000119300389 gpu::CommandBufferService::Flush(int, gpu::AsyncAPIInterface*) + 553 17 Chromium Framework 0x000000011954bf7c gpu::CommandBufferStub::OnAsyncFlush(int, unsigned int) + 364 18 Chromium Framework 0x000000011954bd43 bool IPC::MessageT<GpuCommandBufferMsg_AsyncFlush_Meta, std::__1::tuple<int, unsigned int>, void>::Dispatch<gpu::CommandBufferStub, gpu::CommandBufferStub, void, void (gpu::CommandBufferStub::*)(int, unsigned int)>(IPC::Message const*, gpu::CommandBufferStub*, gpu::CommandBufferStub*, void*, void (gpu::CommandBufferStub::*)(int, unsigned int)) + 115 19 Chromium Framework 0x000000011954a8bb gpu::CommandBufferStub::OnMessageReceived(IPC::Message const&) + 1307 20 Chromium Framework 0x0000000118108a42 IPC::MessageRouter::RouteMessage(IPC::Message const&) + 146 21 Chromium Framework 0x0000000119558d74 gpu::GpuChannel::HandleMessageHelper(IPC::Message const&) + 164 22 Chromium Framework 0x00000001195566b4 gpu::GpuChannel::HandleMessage(IPC::Message const&) + 628 23 Chromium Framework 0x0000000119559e70 base::internal::Invoker<base::internal::BindState<void (gpu::GpuChannel::*)(IPC::Message const&), base::WeakPtr<gpu::GpuChannel>, IPC::Message>, void ()>::RunOnce(base::internal::BindStateBase*) + 192 24 Chromium Framework 0x00000001193074a0 gpu::Scheduler::RunNextTask() + 1200 25 Chromium Framework 0x000000011930a607 base::internal::Invoker<base::internal::BindState<void (gpu::Scheduler::*)(), base::WeakPtr<gpu::Scheduler> >, void ()>::Run(base::internal::BindStateBase*) + 183 26 Chromium Framework 0x00000001176f7466 base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) + 310 27 Chromium Framework 0x00000001177287df base::MessageLoop::RunTask(base::PendingTask*) + 271 28 Chromium Framework 0x0000000117728bd3 base::MessageLoop::DoWork() + 387 29 Chromium Framework 0x000000011772ceca base::MessagePumpCFRunLoopBase::RunWork() + 42 30 Chromium Framework 0x000000011771087a base::mac::CallWithEHFrame(void () block_pointer) + 10 31 Chromium Framework 0x000000011772c7ef base::MessagePumpCFRunLoopBase::RunWorkSource(void*) + 63 32 CoreFoundation 0x00007fff46fa8405 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17 33 CoreFoundation 0x00007fff46fa83ab __CFRunLoopDoSource0 + 108 34 CoreFoundation 0x00007fff46f8be51 __CFRunLoopDoSources0 + 195 35 CoreFoundation 0x00007fff46f8b3fa __CFRunLoopRun + 1219 36 CoreFoundation 0x00007fff46f8ace4 CFRunLoopRunSpecific + 463 37 Foundation 0x00007fff492eb5da -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 280 38 Chromium Framework 0x000000011772d72d base::MessagePumpNSRunLoop::DoRun(base::MessagePump::Delegate*) + 125 39 Chromium Framework 0x000000011772c1fe base::MessagePumpCFRunLoopBase::Run(base::MessagePump::Delegate*) + 110 40 Chromium Framework 0x0000000117728384 base::MessageLoop::Run(bool) + 132 41 Chromium Framework 0x0000000117764639 base::RunLoop::Run() + 249 42 Chromium Framework 0x000000011d792954 content::GpuMain(content::MainFunctionParams const&) + 1444 43 Chromium Framework 0x00000001170e02da content::ContentMainRunnerImpl::Run(bool) + 394 44 Chromium Framework 0x000000011a639a2c service_manager::Main(service_manager::MainParams const&) + 3036 45 Chromium Framework 0x00000001170df454 content::ContentMain(content::ContentMainParams const&) + 68 46 Chromium Framework 0x000000011341493f ChromeMain + 175 47 Chromium Helper 0x000000010733482e main + 494 48 libdyld.dylib 0x00007fff74020085 start + 1 49 ??? 0x000000000000000e 0x0 + 14 The only thing affected by change [1] could be when the path is destroyed. Its now more probable to have a pattern where you create a path and playback on a GPU raster based SkCanvas. But before the GrOpList is flushed, that path is destroyed and could trigger some cleanup logic through SkPathRef::GenIDChangeListener. +csmartdalton@, does the stack and the usage pattern give you any ideas on what could be the bug here? [1]: https://chromium.googlesource.com/chromium/src/+/57df06767601e98a1b51239d3a5f3862e3d000dd
,
Oct 23
Hey jmadill@- You shouldn't have to revert your change. This definitely looks like a bug somewhere else, and you just stepped on the landmine. Do you have steps I can follow to repro? I'd really like to track this down.
,
Oct 23
This wasn't my change actually. It was by khushalsagar. Repro steps are a bit tricky since this only seems to affect Mac 10.14. If you have a Mac with 10.14 installed you should be able to repro by using the commands listed in the swarming task: https://chromium-swarm.appspot.com/task?id=40a91c315e999910&refresh=10&show_raw=1 Namely in this section: Reproducing the task locally Download inputs files into directory foo: # (if needed) git clone https://chromium.googlesource.com/infra/luci/client-py python ./client-py/isolateserver.py download -I https://isolateserver.appspot.com --namespace default-gzip -s a76b54082bba9f23f23d413c31034c83f01d1bcb --target foo Run this task locally: # (if needed) git clone https://chromium.googlesource.com/infra/luci/client-py python ./client-py/swarming.py reproduce -S chromium-swarm.appspot.com 40a91c315e999910 Download output results into directory foo: # (if needed) git clone https://chromium.googlesource.com/infra/luci/client-py python ./client-py/swarming.py collect -S chromium-swarm.appspot.com --task-output-dir=foo 40a91c315e999910 Ken do you have any suggestions for how to repro if csmartdalton doesn't have a 10.14 device? Or any idea when 10.14 will be available on the try bots?
,
Oct 23
Actually, you should be able to log into the machine in question using the steps in the internal GPU pixel wrangling instructions. Internal link but you can find it linked from https://chromium.googlesource.com/chromium/src/+/master/docs/gpu/pixel_wrangling.md
,
Oct 23
Khushal and I looked into this a bit yesterday. The failure was happening on the 10.13 Mac Minis as well: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Release%20%28Intel%29?limit=200 Between this build: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Release%20%28Intel%29/6072 and this one: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Release%20%28Intel%29/6086 We didn't succeed yet in reproducing the failure locally. Will continue to help Khushal to try to get a local repro, which will be easier to deal with than debugging remotely on the bots.
,
Oct 23
I've had pretty good success reproducing failing WebGL tests by using the same arguments as the bots as setting the environment variables GTEST_SHARD_INDEX=## and GTEST_TOTAL_SHARDS=20. Let me know if you want more info on that.
,
Oct 23
The following revision refers to this bug: https://skia.googlesource.com/skia/+/8429c793029135827ca336dee01de7179c683cb4 commit 8429c793029135827ca336dee01de7179c683cb4 Author: Chris Dalton <csmartdalton@google.com> Date: Tue Oct 23 22:31:21 2018 ccpr: Don't use the GrContext id for the path cache id Currently this is probably safe, but somebody could easily make a change in the future that adds another path cache to the context and accidentally introduces tricky bugs. TBR=brianosman@google.com Bug: chromium:897510 Bug: chromium:897413 Bug: chromium:897245 Bug: chromium:897507 Change-Id: I6bc40ac671058f78eb290dd775612d99008d32e7 Reviewed-on: https://skia-review.googlesource.com/c/164700 Reviewed-by: Chris Dalton <csmartdalton@google.com> Commit-Queue: Chris Dalton <csmartdalton@google.com> [modify] https://crrev.com/8429c793029135827ca336dee01de7179c683cb4/src/gpu/ccpr/GrCCAtlas.cpp [modify] https://crrev.com/8429c793029135827ca336dee01de7179c683cb4/src/gpu/ccpr/GrCCDrawPathsOp.cpp [modify] https://crrev.com/8429c793029135827ca336dee01de7179c683cb4/src/gpu/ccpr/GrCCPathCache.h [modify] https://crrev.com/8429c793029135827ca336dee01de7179c683cb4/src/gpu/GrPathRendererChain.cpp [modify] https://crrev.com/8429c793029135827ca336dee01de7179c683cb4/src/gpu/ccpr/GrCoverageCountingPathRenderer.h [modify] https://crrev.com/8429c793029135827ca336dee01de7179c683cb4/src/gpu/ccpr/GrCCAtlas.h [modify] https://crrev.com/8429c793029135827ca336dee01de7179c683cb4/src/gpu/ccpr/GrCoverageCountingPathRenderer_none.cpp [modify] https://crrev.com/8429c793029135827ca336dee01de7179c683cb4/src/gpu/ccpr/GrCCPathCache.cpp [modify] https://crrev.com/8429c793029135827ca336dee01de7179c683cb4/src/gpu/ccpr/GrCoverageCountingPathRenderer.cpp
,
Oct 23
,
Oct 24
The following revision refers to this bug: https://skia.googlesource.com/skia/+/3b57279fd65a91684d5281620f01a25e9639c947 commit 3b57279fd65a91684d5281620f01a25e9639c947 Author: Chris Dalton <csmartdalton@google.com> Date: Wed Oct 24 01:03:41 2018 ccpr: Harden the path cache - Fixes a potential threading issue from freeing the (non-thread-safe) CachedAtlasInfo in the GrCCPathCacheEntry destructor. - Adds logic to HashNode to better guarantee it will remain coherent with the LRU list. - Adds various asserts, including ones to ensure non-thread-safe operations only happen on the graphics thread. TBR=brianosman@google.com Bug: chromium:897510 Bug: chromium:897413 Bug: chromium:897245 Bug: chromium:897507 Change-Id: I5aac8ef712743f62b8c5d0b798804b425084ec02 Reviewed-on: https://skia-review.googlesource.com/c/164707 Reviewed-by: Chris Dalton <csmartdalton@google.com> Commit-Queue: Chris Dalton <csmartdalton@google.com> Auto-Submit: Chris Dalton <csmartdalton@google.com> [modify] https://crrev.com/3b57279fd65a91684d5281620f01a25e9639c947/src/gpu/ccpr/GrCCPathCache.cpp [modify] https://crrev.com/3b57279fd65a91684d5281620f01a25e9639c947/src/gpu/ccpr/GrCCPathCache.h
,
Oct 24
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/86b60a4bc0eee23391beee1ca49a2a65453955e8 commit 86b60a4bc0eee23391beee1ca49a2a65453955e8 Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com> Date: Wed Oct 24 07:54:29 2018 Roll src/third_party/skia 49353683320d..70bd0b72f3f8 (5 commits) https://skia.googlesource.com/skia.git/+log/49353683320d..70bd0b72f3f8 git log 49353683320d..70bd0b72f3f8 --date=short --no-merges --format='%ad %ae %s' 2018-10-24 mtklein@google.com Revert "Some more clean-up of YUVA code" 2018-10-24 csmartdalton@google.com ccpr: Harden the path cache 2018-10-23 recipe-roller@chromium.org Roll recipe dependencies (trivial). 2018-10-23 csmartdalton@google.com ccpr: Don't use the GrContext id for the path cache id 2018-10-23 jvanverth@google.com Some more clean-up of YUVA code Created with: gclient setdep -r src/third_party/skia@70bd0b72f3f8 The AutoRoll server is located here: https://autoroll.skia.org/r/skia-autoroll Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md If the roll is causing failures, please contact the current sheriff, who should be CC'd on the roll, and stop the roller if necessary. CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux-chromeos-compile-dbg;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel;master.tryserver.blink:linux_trusty_blink_rel BUG=chromium:897510,chromium:897413,chromium:897245, chromium:897507 ,chromium:897510,chromium:897413,chromium:897245, chromium:897507 TBR=brianosman@chromium.org Change-Id: I681acc9f101d006ae523d155a7c0975098f166c9 Reviewed-on: https://chromium-review.googlesource.com/c/1297829 Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com> Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com> Cr-Commit-Position: refs/heads/master@{#602273} [modify] https://crrev.com/86b60a4bc0eee23391beee1ca49a2a65453955e8/DEPS
,
Oct 24
csmartdalton@ has been working on adding some asserts and potential fixes for this which it seems like got triggered on the skia bots! I'll reopen and assign to him and reland my patch once the skia fixes are in.
,
Oct 24
Chatted with Chris offline, the skia failure was ddl specific so shouldn't affect chrome. I'll reland this change now and will keep an eye on the bots.
,
Oct 24
You can try using the optional gpu tests bot for Mac on your CL. I think that one tests 10.13.
,
Oct 24
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/02acf1bba75bd6d345b8592c4a9b0ce61e88be56 commit 02acf1bba75bd6d345b8592c4a9b0ce61e88be56 Author: Khushal <khushalsagar@chromium.org> Date: Wed Oct 24 19:59:07 2018 Reland "cc: Cache text blobs in the GPU service." This reverts commit e99280223d68ac7886976c45c562ce4dcbd58d54. Reason for revert: Speculative fixes have landed in skia. Original change's description: > Revert "cc: Cache text blobs in the GPU service." > > This reverts commit 57df06767601e98a1b51239d3a5f3862e3d000dd. > > Reason for revert: Seems to be causing a crash on Mac. > > https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%2010.14%20Release%20%28Intel%29/327 > https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Release%20%28Intel%29/6072 > > Crash in __ZN13GrCCPathCache5evictEP18GrCCPathCacheEntry > > Bug: 897507 > > Original change's description: > > cc: Cache text blobs in the GPU service. > > > > Since skia's internal caching is keyed on text blobs, its important to > > reuse the same object. Make sure we cache text blobs in the > > ServiceTransferCache to get a cache hit in skia. > > > > R=enne@chromium.org > > > > Bug: 894200 > > Change-Id: I8808d649ca3520211d034e07e7d4670ffb39094c > > Reviewed-on: https://chromium-review.googlesource.com/c/1292722 > > Commit-Queue: Khushal <khushalsagar@chromium.org> > > Reviewed-by: enne <enne@chromium.org> > > Cr-Commit-Position: refs/heads/master@{#601404} > > TBR=enne@chromium.org,khushalsagar@chromium.org > > # Not skipping CQ checks because original CL landed > 1 day ago. > > Bug: 894200 > Change-Id: I5e0a151c5a539aa8a9b6cac5f9bcf710ba2f405a > Reviewed-on: https://chromium-review.googlesource.com/c/1292384 > Reviewed-by: Jamie Madill <jmadill@chromium.org> > Commit-Queue: Jamie Madill <jmadill@chromium.org> > Cr-Commit-Position: refs/heads/master@{#601451} TBR=jmadill@chromium.org,enne@chromium.org,khushalsagar@chromium.org # Not skipping CQ checks because original CL landed > 1 day ago. Bug: 897507 , 894200 Change-Id: I345adbbfaff91ab201d68e73e861dd4ccce41775 Reviewed-on: https://chromium-review.googlesource.com/c/1298235 Reviewed-by: Khushal <khushalsagar@chromium.org> Reviewed-by: Jamie Madill <jmadill@chromium.org> Commit-Queue: Khushal <khushalsagar@chromium.org> Cr-Commit-Position: refs/heads/master@{#602406} [modify] https://crrev.com/02acf1bba75bd6d345b8592c4a9b0ce61e88be56/cc/paint/BUILD.gn [modify] https://crrev.com/02acf1bba75bd6d345b8592c4a9b0ce61e88be56/cc/paint/paint_op_reader.cc [modify] https://crrev.com/02acf1bba75bd6d345b8592c4a9b0ce61e88be56/cc/paint/paint_op_writer.cc [add] https://crrev.com/02acf1bba75bd6d345b8592c4a9b0ce61e88be56/cc/paint/textblob_transfer_cache_entry.cc [add] https://crrev.com/02acf1bba75bd6d345b8592c4a9b0ce61e88be56/cc/paint/textblob_transfer_cache_entry.h [modify] https://crrev.com/02acf1bba75bd6d345b8592c4a9b0ce61e88be56/cc/paint/transfer_cache_entry.cc [modify] https://crrev.com/02acf1bba75bd6d345b8592c4a9b0ce61e88be56/cc/paint/transfer_cache_entry.h
,
Oct 24
Looks like the change made it through this time: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Mac%20FYI%20Experimental%20Release%20%28Intel%29/6174. The WebglConformance_conformance_rendering_canvas_alpha_bug failure in that run is a different bug.
,
Oct 24
,
Oct 24
For the record, the canvas-alpha-bug failure is being tracked in Issue 886970.
,
Oct 24
|
||||||||
►
Sign in to add a comment |
||||||||
Comment 1 by jmad...@chromium.org
, Oct 21