EncryptedMediaTest flaky on Linux Asan Lsan |
|||||||||||||||||||
Issue descriptionSee the following flakes: Pepper/ECKEncryptedMediaTest.LoadLoadableSession/0: https://ci.chromium.org/buildbot/chromium.memory/Linux%20ASan%20LSan%20Tests%20%281%29/40743 MSE_ExternalClearKey/EncryptedMediaTest.Playback_VP9Video_WebM_Fullsample/0: https://ci.chromium.org/buildbot/chromium.memory/Linux%20ASan%20LSan%20Tests%20%281%29/40697 Pepper/ECKEncryptedMediaTest.FileIOTest/0: https://ci.chromium.org/buildbot/chromium.memory/Linux%20ASan%20LSan%20Tests%20%281%29/40674 Looks like they are detecting real memory leaks.
,
Dec 9 2017
,
Dec 11 2017
Issue 793921 has been merged into this issue.
,
Dec 11 2017
,
Dec 11 2017
I suspected that some tests are due to my CDM_10 changes (based on the time this started to happen), but it seems some failing tests are using ClearKey (not External Clear Key). For example: https://ci.chromium.org/buildbot/tryserver.chromium.linux/linux_chromium_asan_rel_ng/508822 Assign to jrummell@ to make an asan/lsan build to try it out.
,
Dec 12 2017
,
Dec 12 2017
,
Dec 12 2017
,
Dec 12 2017
Sorry for all the noise! We are looking into it and will get this fixed today.
,
Dec 12 2017
,
Dec 12 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/79c931d227228c798a69857502ccfeca072edae0 commit 79c931d227228c798a69857502ccfeca072edae0 Author: Xiaohan Wang <xhwang@chromium.org> Date: Tue Dec 12 23:18:36 2017 test: Truncate excessive logs in the middle Previously we always truncate logs from the beginning, but there could be important information there. This CL changes the behavior so that we always truncate the middle part of the log to preserve potentially important information in the top and bottom part of the log. BUG=793426 TEST=This improves test infrastructure. Change-Id: I058f735fbaedec324960b2c7a269d549b1a0178e Reviewed-on: https://chromium-review.googlesource.com/823191 Reviewed-by: Dirk Pranke <dpranke@chromium.org> Reviewed-by: Lei Zhang <thestig@chromium.org> Commit-Queue: Xiaohan Wang <xhwang@chromium.org> Cr-Commit-Position: refs/heads/master@{#523596} [modify] https://crrev.com/79c931d227228c798a69857502ccfeca072edae0/base/test/launcher/test_launcher.cc
,
Dec 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/781d87564885f62c352f7cecb2c8717cd87e9c7a commit 781d87564885f62c352f7cecb2c8717cd87e9c7a Author: Xiaohan Wang <xhwang@chromium.org> Date: Wed Dec 13 04:29:51 2017 Reenable flaky EncryptedMediaTest for investigation These tests were flaky and disabled. But we have not been able to repro the issue locally, nor understand the root cause of the issue. Part of the problem is that the top part of the test log is truncated, which has been fixed in a previous CL. Now we need to reproduce the test failure to be able to examine the logs for investigation. This CL reenables those flaky tests by reverting two CLs that disabled those tests: This reverts commit 9f4bee97412177c23c5dd050e2f268306f3b37c8. Revert "Disable ECKEncryptedMediaTest.Renewal on linux asan" This reverts commit b023b4e8692aff593728a98a5faae154e50b3da4. Revert "Disable EncryptedMediaTest.Playback_VideoClearAudio_WebM on Linux asan" TBR=jrummell@chromium.org,cfroussios@chromium.org BUG=793426 Change-Id: I7441ccf1d89ea81d396ba0ab84a77eb41c9bf700 Reviewed-on: https://chromium-review.googlesource.com/823572 Reviewed-by: Xiaohan Wang <xhwang@chromium.org> Commit-Queue: Xiaohan Wang <xhwang@chromium.org> Cr-Commit-Position: refs/heads/master@{#523688} [modify] https://crrev.com/781d87564885f62c352f7cecb2c8717cd87e9c7a/chrome/browser/media/encrypted_media_browsertest.cc
,
Dec 13 2017
Issue 794228 has been merged into this issue.
,
Dec 13 2017
Issue 794227 has been merged into this issue.
,
Dec 13 2017
Issue 794198 has been merged into this issue.
,
Dec 13 2017
Issue 794197 has been merged into this issue.
,
Dec 13 2017
,
Dec 13 2017
+meade The tests failed multiple times over night. And I can see the top of the log. It seems the test itself finished successfully, then it detected memory leak, in the renderer process, and the first "Direct leak" is always related to blink::CSSVariableParser::ParseDeclarationValue. meade: Is it possible that this is related to https://chromium-review.googlesource.com/c/chromium/src/+/790171 ? Example log: https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.memory%2FLinux_ASan_LSan_Tests__1_%2F40854%2F%2B%2Frecipes%2Fsteps%2Fbrowser_tests%2F0%2Flogs%2FMSE_ExternalClearKey__x2f_EncryptedMediaTest.FrameSizeChangeVideo__x2f_0%2F0 .... [24018:24018:1213/061710.753432:INFO:CONSOLE(277)] "06:17:10.750 - One video seeked.", source: http://127.0.0.1:53123/eme_player_js/utils.js (277) [24018:24018:1213/061710.867861:INFO:CONSOLE(277)] "06:17:10.812 - Set document title to: ENDED, updated title: ENDED", source: http://127.0.0.1:53123/eme_player_js/utils.js (277) ================================================================= ==24170==ERROR: LeakSanitizer: detected memory leaks Direct leak of 35672 byte(s) in 343 object(s) allocated from: #0 0x79fbd33 in __interceptor_malloc /b/build/slave/linux_upload_clang/build/src/third_party/llvm/compiler-rt/lib/asan/asan_malloc_linux.cc:88:3 #1 0xd2860c9 in PartitionAllocGenericFlags base/allocator/partition_allocator/partition_alloc.h:942:18 #2 0xd2860c9 in Alloc base/allocator/partition_allocator/partition_alloc.h:962 #3 0xd2860c9 in WTF::Partitions::FastMalloc(unsigned long, char const*) third_party/WebKit/Source/platform/wtf/allocator/Partitions.h:121 #4 0x1f9023d6 in operator new third_party/WebKit/Source/core/css/CSSVariableData.h:21:3 #5 0x1f9023d6 in Create third_party/WebKit/Source/core/css/CSSVariableData.h:27 #6 0x1f9023d6 in blink::CSSVariableParser::ParseDeclarationValue(WTF::AtomicString const&, blink::CSSParserTokenRange, bool) third_party/WebKit/Source/core/css/parser/CSSVariableParser.cpp:132 #7 0x1f88abc0 in blink::CSSParserImpl::ConsumeVariableValue(blink::CSSParserTokenRange, WTF::AtomicString const&, bool, bool) third_party/WebKit/Source/core/css/parser/CSSParserImpl.cpp:971:11 #8 0x1f898858 in blink::CSSParserImpl::ConsumeDeclaration(blink::CSSParserTokenRange, blink::CSSParserImpl::RangeOffset const&, blink::StyleRuleBase::RuleType) third_party/WebKit/Source/core/css/parser/CSSParserImpl.cpp:947:5 #9 0x1f88c656 in blink::CSSParserImpl::ConsumeDeclarationList(blink::CSSParserTokenStream&, blink::StyleRuleBase::RuleType) third_party/WebKit/Source/core/css/parser/CSSParserImpl.cpp:876:9 #10 0x1f8a1d4d in blink::CSSParserImpl::ConsumeStyleRule(blink::CSSParserTokenStream&) third_party/WebKit/Source/core/css/parser/CSSParserImpl.cpp:828:3 #11 0x1f8920e7 in blink::CSSParserImpl::ConsumeQualifiedRule(blink::CSSParserTokenStream&, blink::CSSParserImpl::AllowedRulesType) third_party/WebKit/Source/core/css/parser/CSSParserImpl.cpp:529:12 #12 0x1f89312f in ConsumeRuleList<(lambda at ../../third_party/WebKit/Source/core/css/parser/CSSParserImpl.cpp:264:34)> third_party/WebKit/Source/core/css/parser/CSSParserImpl.cpp:442:16 #13 0x1f89312f in blink::CSSParserImpl::ParseStyleSheet(WTF::String const&, blink::CSSParserContext const*, blink::StyleSheetContents*, bool) third_party/WebKit/Source/core/css/parser/CSSParserImpl.cpp:263 #14 0x1f6a6b6c in ParseSheet third_party/WebKit/Source/core/css/StyleEngine.cpp:656:28 #15 0x1f6a6b6c in blink::StyleEngine::CreateSheet(blink::Element&, WTF::String const&, WTF::TextPosition, blink::StyleEngineContext&) third_party/WebKit/Source/core/css/StyleEngine.cpp:625 #16 0x20b22b56 in blink::StyleElement::CreateSheet(blink::Element&, WTF::String const&) third_party/WebKit/Source/core/css/StyleElement.cpp:163:43 #17 0x20b22284 in Process third_party/WebKit/Source/core/css/StyleElement.cpp:109:10 #18 0x20b22284 in blink::StyleElement::FinishParsingChildren(blink::Element&) third_party/WebKit/Source/core/css/StyleElement.cpp:101 #19 0x20b1e008 in blink::HTMLStyleElement::FinishParsingChildren() third_party/WebKit/Source/core/html/HTMLStyleElement.cpp:70:21 #20 0x20bb12a3 in blink::HTMLElementStack::PopCommon() third_party/WebKit/Source/core/html/parser/HTMLElementStack.cpp:501:10 #21 0x20bb25e8 in blink::HTMLElementStack::Pop() third_party/WebKit/Source/core/html/parser/HTMLElementStack.cpp:179:3 #22 0x20cc6a3e in blink::HTMLTreeBuilder::ProcessEndTag(blink::AtomicHTMLToken*) third_party/WebKit/Source/core/html/parser/HTMLTreeBuilder.cpp:2050:29 #23 0x20caae2b in blink::HTMLTreeBuilder::ProcessToken(blink::AtomicHTMLToken*) third_party/WebKit/Source/core/html/parser/HTMLTreeBuilder.cpp:359:7 #24 0x20ca2d86 in blink::HTMLTreeBuilder::ConstructTree(blink::AtomicHTMLToken*) third_party/WebKit/Source/core/html/parser/HTMLTreeBuilder.cpp:313:5 #25 0x20b39340 in ConstructTreeFromCompactHTMLToken third_party/WebKit/Source/core/html/parser/HTMLDocumentParser.cpp:747:18 #26 0x20b39340 in blink::HTMLDocumentParser::ProcessTokenizedChunkFromBackgroundParser(std::__1::unique_ptr<blink::HTMLDocumentParser::TokenizedChunk, std::__1::default_delete<blink::HTMLDocumentParser::TokenizedChunk> >) third_party/WebKit/Source/core/html/parser/HTMLDocumentParser.cpp:534 #27 0x20b318ea in blink::HTMLDocumentParser::PumpPendingSpeculations() third_party/WebKit/Source/core/html/parser/HTMLDocumentParser.cpp:608:9 #28 0x20b30a9e in blink::HTMLDocumentParser::ResumeParsingAfterYield() third_party/WebKit/Source/core/html/parser/HTMLDocumentParser.cpp:273:3 #29 0x1259a733 in Run base/callback.h:65:12 #30 0x1259a733 in RunInternal third_party/WebKit/Source/platform/wtf/Functional.h:258 #31 0x1259a733 in WTF::ThreadCheckingCallbackWrapper<base::OnceCallback<void ()>, void ()>::Run() third_party/WebKit/Source/platform/wtf/Functional.h:245 #32 0x1ea294c9 in Run base/callback.h:65:12 #33 0x1ea294c9 in blink::TaskHandle::Runner::Run(blink::TaskHandle const&) third_party/WebKit/Source/platform/WebTaskRunner.cpp:75 #34 0x1ea2a725 in Invoke<base::WeakPtr<blink::TaskHandle::Runner>, blink::TaskHandle> base/bind_internal.h:211:12 #35 0x1ea2a725 in MakeItSo<void (blink::TaskHandle::Runner::*)(const blink::TaskHandle &), base::WeakPtr<blink::TaskHandle::Runner>, blink::TaskHandle> base/bind_internal.h:314 #36 0x1ea2a725 in RunImpl<void (blink::TaskHandle::Runner::*)(const blink::TaskHandle &), std::__1::tuple<base::WeakPtr<blink::TaskHandle::Runner>, blink::TaskHandle>, 0, 1> base/bind_internal.h:368 #37 0x1ea2a725 in base::internal::Invoker<base::internal::BindState<void (blink::TaskHandle::Runner::*)(blink::TaskHandle const&), base::WeakPtr<blink::TaskHandle::Runner>, blink::TaskHandle>, void ()>::RunOnce(base::internal::BindStateBase*) base/bind_internal.h:336 #38 0x1259a733 in Run base/callback.h:65:12 #39 0x1259a733 in RunInternal third_party/WebKit/Source/platform/wtf/Functional.h:258 #40 0x1259a733 in WTF::ThreadCheckingCallbackWrapper<base::OnceCallback<void ()>, void ()>::Run() third_party/WebKit/Source/platform/wtf/Functional.h:245 #41 0x12f6dc2b in Run base/callback.h:65:12 #42 0x12f6dc2b in base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) base/debug/task_annotator.cc:55 #43 0x126c01c8 in blink::scheduler::TaskQueueManager::ProcessTaskFromWorkQueue(blink::scheduler::internal::WorkQueue*, bool, blink::scheduler::LazyNow, base::TimeTicks*) third_party/WebKit/Source/platform/scheduler/base/task_queue_manager.cc:513:21 #44 0x126be940 in blink::scheduler::TaskQueueManager::DoWork(blink::scheduler::internal::Sequence::WorkType) third_party/WebKit/Source/platform/scheduler/base/task_queue_manager.cc:320:13 #45 0x126cdc0a in Invoke<const base::WeakPtr<blink::scheduler::TaskQueueManager> &, const blink::scheduler::internal::Sequence::WorkType &> base/bind_internal.h:211:12 #46 0x126cdc0a in MakeItSo<void (blink::scheduler::TaskQueueManager::*const &)(blink::scheduler::internal::Sequence::WorkType), const base::WeakPtr<blink::scheduler::TaskQueueManager> &, const blink::scheduler::internal::Sequence::WorkType &> base/bind_internal.h:314 #47 0x126cdc0a in RunImpl<void (blink::scheduler::TaskQueueManager::*const &)(blink::scheduler::internal::Sequence::WorkType), const std::__1::tuple<base::WeakPtr<blink::scheduler::TaskQueueManager>, blink::scheduler::internal::Sequence::WorkType> &, 0, 1> base/bind_internal.h:368 #48 0x126cdc0a in base::internal::Invoker<base::internal::BindState<void (blink::scheduler::TaskQueueManager::*)(blink::scheduler::internal::Sequence::WorkType), base::WeakPtr<blink::scheduler::TaskQueueManager>, blink::scheduler::internal::Sequence::WorkType>, void ()>::Run(base::internal::BindStateBase*) base/bind_internal.h:350 #49 0x12f6dc2b in Run base/callback.h:65:12 #50 0x12f6dc2b in base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask*) base/debug/task_annotator.cc:55
,
Dec 13 2017
,
Dec 13 2017
Assign to myself to keep track of this issue. Originally I suspected that this could be caused by my recent CL: https://chromium-review.googlesource.com/813087 But that CL was landed on Dec 7, and the earliest failure was on Dec 5: https://ci.chromium.org/buildbot/chromium.memory/Linux%20ASan%20LSan%20Tests%20%281%29/40674 Also, my CL should not affect the renderer process (the leak is in Blink). The last CL that touches blink::CSSVariableParser::ParseDeclarationValue was https://chromium-review.googlesource.com/c/chromium/src/+/790171 which was landed on Nov 28, so at least the timing makes some sense.
,
Dec 13 2017
xhwang@ If you're having trouble reproing the issue locally, you can grab a bot to test on. https://chrome-internal.googlesource.com/infra/infra_internal/+/master/doc/swarming_ssh.md
,
Dec 13 2017
I looked at a different log: https://logs.chromium.org/v/?s=chromium%2Fbb%2Ftryserver.chromium.linux%2Flinux_chromium_asan_rel_ng%2F510240%2F%2B%2Frecipes%2Fsteps%2Fbrowser_tests__with_patch_%2F0%2Flogs%2FMSE_ClearKey__x2f_EncryptedMediaTest.Playback_AudioClearVideo_WebM__x2f_0%2F0 Most (all?) of the leaks are generated when resources are loaded. None reference any media code (search for media or encrypted doesn't show up in any callstack). Truncated traces from the first 5 reported leaks. Direct leak of 35672 byte(s) in 343 object(s) allocated from: #26 0x20b21bc0 in blink::HTMLDocumentParser::ProcessTokenizedChunkFromBackgroundParser(...) third_party/WebKit/Source/core/html/parser/HTMLDocumentParser.cpp:534 #27 0x20b1a16a in blink::HTMLDocumentParser::PumpPendingSpeculations() third_party/WebKit/Source/core/html/parser/HTMLDocumentParser.cpp:608:9 #28 0x20b1931e in blink::HTMLDocumentParser::ResumeParsingAfterYield() third_party/WebKit/Source/core/html/parser/HTMLDocumentParser.cpp:273:3 Direct leak of 4992 byte(s) in 20 object(s) allocated from: #20 0x24be4e90 in content::WebURLLoaderImpl::Context::OnReceivedResponse(...) content/renderer/loader/web_url_loader_impl.cc:820:12 #21 0x24be7f19 in content::WebURLLoaderImpl::RequestPeerImpl::OnReceivedResponse(...) content/renderer/loader/web_url_loader_impl.cc:1052:13 #22 0x248d3db4 in content::ResourceDispatcher::OnReceivedResponse(...) content/renderer/loader/resource_dispatcher.cc:228:23 #23 0x248f24a4 in content::URLLoaderClientImpl::OnReceiveResponse(...) content/renderer/loader/url_loader_client_impl.cc:123:27 #24 0xc5db5f2 in content::ThrottlingURLLoader::OnReceiveResponse(...) content/common/throttling_url_loader.cc:350:23 Direct leak of 4833 byte(s) in 9 object(s) allocated from: #9 0x20b6e301 in blink::CompactHTMLToken::CompactHTMLToken(...) third_party/WebKit/Source/core/html/parser/CompactHTMLToken.cpp:79 #10 0x20b6320b in blink::BackgroundHTMLParser::PumpTokenizer() third_party/WebKit/Source/core/html/parser/BackgroundHTMLParser.cpp:257:24 #11 0x20b626b6 in AppendDecodedBytes third_party/WebKit/Source/core/html/parser/BackgroundHTMLParser.cpp:148:3 #12 0x20b626b6 in blink::BackgroundHTMLParser::UpdateDocument(...) third_party/WebKit/Source/core/html/parser/BackgroundHTMLParser.cpp:177 #13 0x20b6229c in blink::BackgroundHTMLParser::AppendRawBytesFromMainThread(...) third_party/WebKit/Source/core/html/parser/BackgroundHTMLParser.cpp:142:3 Direct leak of 3584 byte(s) in 15 object(s) allocated from: #19 0x24be4e90 in content::WebURLLoaderImpl::Context::OnReceivedResponse(...) content/renderer/loader/web_url_loader_impl.cc:820:12 #20 0x24be7f19 in content::WebURLLoaderImpl::RequestPeerImpl::OnReceivedResponse(...) content/renderer/loader/web_url_loader_impl.cc:1052:13 #21 0x248d3db4 in content::ResourceDispatcher::OnReceivedResponse(...) content/renderer/loader/resource_dispatcher.cc:228:23 #22 0x248f24a4 in content::URLLoaderClientImpl::OnReceiveResponse(...) content/renderer/loader/url_loader_client_impl.cc:123:27 #23 0xc5db5f2 in content::ThrottlingURLLoader::OnReceiveResponse(...) content/common/throttling_url_loader.cc:350:23 Direct leak of 2720 byte(s) in 34 object(s) allocated from: #8 0x24be44b6 in content::WebURLLoaderImpl::Context::OnReceivedResponse(...) content/renderer/loader/web_url_loader_impl.cc:771:3 #9 0x24be7f19 in content::WebURLLoaderImpl::RequestPeerImpl::OnReceivedResponse(...) content/renderer/loader/web_url_loader_impl.cc:1052:13 #10 0x248d3db4 in content::ResourceDispatcher::OnReceivedResponse(...) content/renderer/loader/resource_dispatcher.cc:228:23 #11 0x248f24a4 in content::URLLoaderClientImpl::OnReceiveResponse(...) content/renderer/loader/url_loader_client_impl.cc:123:27 #12 0xc5db5f2 in content::ThrottlingURLLoader::OnReceiveResponse(...) content/common/throttling_url_loader.cc:350:23 I assume from the above that this means that the test DOM/JavaScript is not being cleaned up properly when the test exits (although only occasionally, as most of the runs do succeed). The test that runs most of the encrypted media browser tests (https://cs.chromium.org/chromium/src/media/test/data/eme_player.html) does load 11 JavaScript files. However, there have been no significant changes to the test in 2017.
,
Dec 13 2017
+shend@ who reviewed that CL since meade@ is OOO today.
,
Dec 13 2017
I don't think meade's CL is the cause. It just removes dead code in a feature that was never shipped.
,
Dec 13 2017
Thanks! Then do you have any idea how blink::CSSVariableParser::ParseDeclarationValue() can leak memory?
,
Dec 13 2017
Sorry, nothing comes to mind :( So I'm trying to understand the stack trace containing CSSVariableParser::ParseDeclarationValue(). So an object leaked because it was allocated but never freed? And is the stack trace showing where that object was originally allocated? If so, then it looks like a CSSVariableData object was created in the parser and was never deallocated. But that seems weird since CSSVariableData is ref counted, so the only possibility I can think of is something with a long lifetime is holding a reference to a CSSVariableData (directly or indirectly). It would be nice to be able to see the refcount of that CSSVariableData when the test finished or even what objects are holding those references.
,
Dec 13 2017
jrummell@ pointed to me that in the log, those leaked memory entries are ordered by the number of bytes leaked, and blink::CSSVariableParser::ParseDeclarationValue() is on the top because it leaked the most number of bytes. Given ASAN didn't catch anything, I assume there's no memory corruption. So it seems to me that this is related to how allocated memory is managed. +ajwong who refactored PartitionAlloc API recently (see issue 787153) for any insight.
,
Dec 13 2017
Almost all my changes were function or variable renames... I don't believe it's likely to have caused a leak?
,
Dec 13 2017
That makes sense, just that your CLs do appear in the blamelist shortly before the first failing build. I'll keep looking.
,
Dec 14 2017
+glider for insights. Can this be the same problem as issue 586897 where V8 doesn't finish GC before the test ends?
,
Dec 15 2017
This is still happening. https://ci.chromium.org/buildbot/chromium.memory/Linux%20ASan%20LSan%20Tests%20%281%29/40939 +nhiroki@ for insights from blink side.
,
Dec 15 2017
,
Dec 18 2017
xhwang@ is there any way I can help debug this issue? Personally, I would recommend grabbing a swarming bot to test on, and doing a git bisect. It will be tricky because you'll need to re-run the test several times per step, but I think it will work.
,
Dec 19 2017
thomasanderson@: I can give it a try when I have time. But giving how flaky this test is, I am not sure whether bi-sect will work well (never tried bi-sect on flaky tests). Also, see #30, I still suspect this is related to V8 not finishing GC properly when test ends (similar to issue 586897). So I was hoping to get some insights from Blink owners, but so far I haven't heard anything yet.
,
Dec 21 2017
FYI I haven't seen any flakes in the past ~150 runs of Linux Asan Lsan Tests, so this issue might be fixed.
,
Dec 28 2017
,
Jan 2 2018
Re #35: Unfortunately it is still happening. In the last 200 runs it reproed twice: https://ci.chromium.org/buildbot/chromium.memory/Linux%20ASan%20LSan%20Tests%20%281%29/?limit=200 nhiroki: kingly ping! Please let me know who'd be the best owner on the Blink side to talk about potential issue of GC not being run during test tear down.
,
Jan 3 2018
,
Feb 12 2018
There's 1 flakiness in the last 800 runs: https://logs.chromium.org/v/?s=chromium%2Fbb%2Fchromium.memory%2FLinux_ASan_LSan_Tests__1_%2F42388%2F%2B%2Frecipes%2Fsteps%2Fbrowser_tests%2F0%2Flogs%2FMSE_ExternalClearKey_Mojo__x2f_EncryptedMediaTest.Playback_VideoAudio_WebM_Opus__x2f_0%2F0 Also, I feel it's more related to the Blink side GC vs testing. Some help from the blink owners would be appreciated. With that, I'll mark this as P2.
,
Jun 7 2018
,
Jun 7 2018
Another Asan flake for ECKEncryptedMediaTest.LoadLoadableSession: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_asan_rel_ng/29701
,
Jun 8 2018
None of the leaks reported in the run from #41 involve media code. Top leak is from CSS [1]. Test doesn't use any CSS, so no idea why ASAN is reporting leaks allocated this way. Where does the CSS come from?
Looking further down the list (randomly), there are a bunch of task related leaks (example [2]). If I look at the code referenced (task_queue_with_task_type.cc:20), the code creates a ref counted object:
return base::WrapRefCounted(
new TaskQueueWithTaskType(std::move(task_queue), task_type));
So is the system doing a quick shut down and leaving pending tasks? Is there any way for the test to make sure everything is done before quitting?
[1]
Direct leak of 35776 byte(s) in 344 object(s) allocated from:
#0 0x839d123 in __interceptor_malloc /b/build/slave/linux_upload_clang/build/src/third_party/llvm/compiler-rt/lib/asan/asan_malloc_linux.cc:98:3
#1 0x12375f5d in PartitionAllocGenericFlags base/allocator/partition_allocator/partition_alloc.h:318:18
#2 0x12375f5d in Alloc base/allocator/partition_allocator/partition_alloc.h:338
#3 0x12375f5d in WTF::Partitions::FastMalloc(unsigned long, char const*) third_party/blink/renderer/platform/wtf/allocator/partitions.h:121
#4 0x22087e2a in operator new third_party/blink/renderer/core/css/css_variable_data.h:21:3
#5 0x22087e2a in Create third_party/blink/renderer/core/css/css_variable_data.h:30
#6 0x22087e2a in blink::CSSVariableParser::ParseDeclarationValue(WTF::AtomicString const&, blink::CSSParserTokenRange, bool) third_party/blink/renderer/core/css/parser/css_variable_parser.cc:165
#7 0x21ffd8e6 in blink::CSSParserImpl::ConsumeVariableValue(blink::CSSParserTokenRange, WTF::AtomicString const&, bool, bool) third_party/blink/renderer/core/css/parser/css_parser_impl.cc:970:11
#8 0x2200b182 in blink::CSSParserImpl::ConsumeDeclaration(blink::CSSParserTokenRange, blink::CSSParserImpl::RangeOffset const&, blink::StyleRuleBase::RuleType) third_party/blink/renderer/core/css/parser/css_parser_impl.cc:946:5
#9 0x21fff3ec in blink::CSSParserImpl::ConsumeDeclarationList(blink::CSSParserTokenStream&, blink::StyleRuleBase::RuleType) third_party/blink/renderer/core/css/parser/css_parser_impl.cc:875:9
[2]
Direct leak of 416 byte(s) in 13 object(s) allocated from:
#0 0x83c8f72 in operator new(unsigned long) /b/build/slave/linux_upload_clang/build/src/third_party/llvm/compiler-rt/lib/asan/asan_new_delete.cc:93:3
#1 0x1490ffb0 in blink::scheduler::TaskQueueWithTaskType::Create(scoped_refptr<base::sequence_manager::TaskQueue>, blink::TaskType) third_party/blink/renderer/platform/scheduler/child/task_queue_with_task_type.cc:20:7
#2 0x149488a7 in blink::scheduler::FrameSchedulerImpl::GetTaskRunner(blink::TaskType) third_party/blink/renderer/platform/scheduler/main_thread/frame_scheduler_impl.cc:278:14
,
Jun 8 2018
linux_chromium_asan_rel_ng is in really bad shape because of this bug. https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_asan_rel_ng?limit=200 Looks like a good 30% of all runs are false positives in viz_browser_tests/EncryptedMediaTest
,
Jun 8 2018
FYI, since that bot is CQ-blocking, it is blocking devs. Hence why I raised the priority.
,
Jun 8 2018
See #34, I still feel the best way forward is to get blink/v8 folks involved.
,
Jun 8 2018
thomasanderson: Maybe we can disable all EncryptedMediaTest on Linux Asan build so solve the CQ-blocking issue. WDYT?
,
Jun 8 2018
re c#46 sgtm
,
Jun 8 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/02f56ede4d3e34f963dd34285701d066cc12aef2 commit 02f56ede4d3e34f963dd34285701d066cc12aef2 Author: Xiaohan Wang <xhwang@chromium.org> Date: Fri Jun 08 06:37:58 2018 media: Disable chrome EncryptedMediaTest on Linux Asan Lsan These tests are flaky on this bot. Disable them for now so that we do not block the CQ. Bug: 793426 Test: Disable tests Change-Id: I12cd105bd2c57a44156a04e33f1a456f4492a996 Reviewed-on: https://chromium-review.googlesource.com/1092311 Reviewed-by: Thomas Anderson <thomasanderson@chromium.org> Commit-Queue: Xiaohan Wang <xhwang@chromium.org> Cr-Commit-Position: refs/heads/master@{#565572} [modify] https://crrev.com/02f56ede4d3e34f963dd34285701d066cc12aef2/chrome/browser/media/encrypted_media_browsertest.cc [modify] https://crrev.com/02f56ede4d3e34f963dd34285701d066cc12aef2/chrome/test/BUILD.gn
,
Jun 8 2018
Issue 850817 has been merged into this issue.
,
Jun 8 2018
These tests have been disabled on the linux asan/lsan bot and recent builds look good to me: https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_asan_rel_ng?limit=250 Downgrade to P1 since this is not blocking CQ anymore.
,
Jun 8 2018
Issue 851019 has been merged into this issue.
,
Jul 19
,
Nov 2
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/a0d3e6638a1639d420f30c954b24ce41e8fd06de commit a0d3e6638a1639d420f30c954b24ce41e8fd06de Author: Evan Stade <estade@chromium.org> Date: Fri Nov 02 19:31:43 2018 Make sure all single process mash browser test filters have bug links. Also, - Enable a couple locally passing tests that don't have bugs attached. - Disable EncryptedMediaTest suite on Chrome OS asan/lsan, to match Linux asan/lsan, due to flake (not unique to Mash). Bug: 793426 Change-Id: Ia01b402234e33e5fda0f587fc8d47fad7897cbb7 Reviewed-on: https://chromium-review.googlesource.com/c/1315809 Commit-Queue: Evan Stade <estade@chromium.org> Reviewed-by: James Cook <jamescook@chromium.org> Reviewed-by: Xiaohan Wang <xhwang@chromium.org> Cr-Commit-Position: refs/heads/master@{#605015} [modify] https://crrev.com/a0d3e6638a1639d420f30c954b24ce41e8fd06de/chrome/test/BUILD.gn [modify] https://crrev.com/a0d3e6638a1639d420f30c954b24ce41e8fd06de/testing/buildbot/filters/chromeos.single_process_mash.browser_tests.filter |
|||||||||||||||||||
►
Sign in to add a comment |
|||||||||||||||||||
Comment 1 by xhw...@chromium.org
, Dec 8 2017Labels: M-65
They all look like they are leaking in some blink code which I have no idea about what's happening: Direct leak of 8 byte(s) in 1 object(s) allocated from: #0 0x79b8c82 in operator new(unsigned long) /b/build/slave/linux_upload_clang/build/src/third_party/llvm/compiler-rt/lib/asan/asan_new_delete.cc:92:3 #1 0x1ecf592a in Create third_party/WebKit/Source/platform/animation/CompositorAnimationTimeline.h:25:28 #2 0x1ecf592a in blink::DocumentTimeline::DocumentTimeline(blink::Document*, double, blink::DocumentTimeline::PlatformTiming*) third_party/WebKit/Source/core/animation/DocumentTimeline.cpp:91 #3 0x1ecf50b8 in blink::DocumentTimeline::Create(blink::Document*, double, blink::DocumentTimeline::PlatformTiming*) third_party/WebKit/Source/core/animation/DocumentTimeline.cpp:65:14 #4 0x1f8ece94 in blink::Document::Document(blink::DocumentInit const&, unsigned char) third_party/WebKit/Source/core/dom/Document.cpp:646:17 #5 0x206ff258 in blink::HTMLDocument::HTMLDocument(blink::DocumentInit const&, unsigned char) third_party/WebKit/Source/core/html/HTMLDocument.cpp:67:7 #6 0x1f8c7e42 in Create third_party/WebKit/Source/core/html/HTMLDocument.h:37:16 #7 0x1f8c7e42 in blink::DOMImplementation::createDocument(WTF::String const&, blink::DocumentInit const&, bool) third_party/WebKit/Source/core/dom/DOMImplementation.cpp:233 #8 0x20301ba8 in blink::LocalDOMWindow::CreateDocument(WTF::String const&, blink::DocumentInit const&, bool) third_party/WebKit/Source/core/frame/LocalDOMWindow.cpp:298:16 #9 0x203020ac in blink::LocalDOMWindow::InstallNewDocument(WTF::String const&, blink::DocumentInit const&, bool) third_party/WebKit/Source/core/frame/LocalDOMWindow.cpp:320:15 #10 0x21abf480 in blink::DocumentLoader::InstallNewDocument(blink::KURL const&, blink::Document*, bool, WTF::AtomicString const&, WTF::AtomicString const&, blink::DocumentLoader::InstallNewDocumentReason, blink::ParserSynchronizationPolicy, blink::KURL const&) third_party/WebKit/Source/core/loader/DocumentLoader.cpp:1088:45 #11 0x21abeb00 in blink::DocumentLoader::CommitNavigation(WTF::AtomicString const&, blink::KURL const&) third_party/WebKit/Source/core/loader/DocumentLoader.cpp:687:3 #12 0x21abae66 in blink::DocumentLoader::CommitData(char const*, unsigned long) third_party/WebKit/Source/core/loader/DocumentLoader.cpp:697:3 #13 0x21ab9bf8 in blink::DocumentLoader::FinishedLoading(double) third_party/WebKit/Source/core/loader/DocumentLoader.cpp:454:7 #14 0x21ac0fad in blink::DocumentLoader::MaybeLoadEmpty() third_party/WebKit/Source/core/loader/DocumentLoader.cpp:841:3 #15 0x21ac1444 in blink::DocumentLoader::StartLoading() third_party/WebKit/Source/core/loader/DocumentLoader.cpp:851:7 #16 0x21b17a3c in blink::FrameLoader::Init() third_party/WebKit/Source/core/loader/FrameLoader.cpp:284:33 #17 0x20358e0b in blink::LocalFrame::Init() third_party/WebKit/Source/core/frame/LocalFrame.cpp:148:11 #18 0x205415bc in blink::WebLocalFrameImpl::InitializeCoreFrame(blink::Page&, blink::FrameOwner*, WTF::AtomicString const&) third_party/WebKit/Source/core/frame/WebLocalFrameImpl.cpp:1757:11 #19 0x20540568 in blink::WebLocalFrameImpl::CreateMainFrame(blink::WebView*, blink::WebFrameClient*, blink::InterfaceRegistry*, blink::WebFrame*, blink::WebString const&, blink::WebSandboxFlags) third_party/WebKit/Source/core/frame/WebLocalFrameImpl.cpp:1635:10 #20 0x245a3db9 in content::RenderFrameImpl::CreateMainFrame(content::RenderViewImpl*, int, mojo::InterfacePtr<service_manager::mojom::InterfaceProvider>, int, bool, content::ScreenInfo const&, content::CompositorDependencies*, blink::WebFrame*, base::UnguessableToken const&, content::FrameReplicationState const&) content/renderer/render_frame_impl.cc:1096:30 #21 0x24afbeb2 in content::RenderViewImpl::Initialize(mojo::StructPtr<content::mojom::CreateViewParams>, base::RepeatingCallback<void (content::RenderWidget*, blink::WebNavigationPolicy, gfx::Rect const&)> const&) content/renderer/render_view_impl.cc:607:26 #22 0x24b0423a in content::RenderViewImpl::Create(content::CompositorDependencies*, mojo::StructPtr<content::mojom::CreateViewParams>, base::RepeatingCallback<void (content::RenderWidget*, blink::WebNavigationPolicy, gfx::Rect const&)> const&, scoped_refptr<base::SingleThreadTaskRunner>) content/renderer/render_view_impl.cc:1024:16 #23 0x249b12a6 in content::RenderThreadImpl::CreateView(mojo::StructPtr<content::mojom::CreateViewParams>) content/renderer/render_thread_impl.cc:2219:3 #24 0xc5d2afb in content::mojom::RendererStubDispatch::Accept(content::mojom::Renderer*, mojo::Message*) out/Release/gen/content/common/renderer.mojom.cc:633:13 #25 0x17065d53 in mojo::InterfaceEndpointClient::HandleValidatedMessage(mojo::Message*) mojo/public/cpp/bindings/lib/interface_endpoint_client.cc:418:32 #26 0x1706ef47 in mojo::FilterChain::Accept(mojo::Message*) mojo/public/cpp/bindings/lib/filter_chain.cc:40:17 #27 0x1706a197 in mojo::InterfaceEndpointClient::HandleIncomingMessage(mojo::Message*) mojo/public/cpp/bindings/lib/interface_endpoint_client.cc:305:19 #28 0x171561ee in IPC::(anonymous namespace)::ChannelAssociatedGroupController::AcceptOnProxyThread(mojo::Message) ipc/ipc_mojo_bootstrap.cc:789:24 #29 0x1714f248 in Invoke<const scoped_refptr<IPC::(anonymous namespace)::ChannelAssociatedGroupController> &, mojo::Message> base/bind_internal.h:211:12 #30 0x1714f248 in MakeItSo<void (IPC::(anonymous namespace)::ChannelAssociatedGroupController::*const &)(mojo::Message), const scoped_refptr<IPC::(anonymous namespace)::ChannelAssociatedGroupController> &, mojo::Message> base/bind_internal.h:294 #31 0x1714f248 in RunImpl<void (IPC::(anonymous namespace)::ChannelAssociatedGroupController::*const &)(mojo::Message), const std::__1::tuple<scoped_refptr<IPC::(anonymous namespace)::ChannelAssociatedGroupController>, base::internal::PassedWrapper<mojo::Message> > &, 0, 1> base/bind_internal.h:368 #32 0x1714f248 in base::internal::Invoker<base::internal::BindState<void (IPC::(anonymous namespace)::ChannelAssociatedGroupController::*)(mojo::Message), scoped_refptr<IPC::(anonymous namespace)::ChannelAssociatedGroupController>, base::internal::PassedWrapper<mojo::Message> >, void ()>::Run(base::internal::BindStateBase*) base/bind_internal.h:350 #33 0x12c69a0b in Run base/callback.h:65:12