[viz_]browser_tests WebRtcBrowserTest leaks detected |
|||||||||||||||
Issue descriptionFiled by sheriff-o-matic@appspot.gserviceaccount.com on behalf of dalecurtis@chromium.org viz_browser_tests failing on chromium.memory/Linux Chromium OS ASan LSan Tests (1) Builders failed on: - Linux Chromium OS ASan LSan Tests (1): https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Chromium%20OS%20ASan%20LSan%20Tests%20%281%29 Many WebRTC tests are leaking page level things like a UkmRecorder and security origin:
,
Oct 25
Failing builds: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Chromium%20OS%20ASan%20LSan%20Tests%20%281%29/29738 WebRtcBrowserTest.RunsAudioVideoWebRTCCallInTwoTabsOfferEcdsaAnswerEcdsa https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Chromium%20OS%20ASan%20LSan%20Tests%20%281%29/29736 WebRtcBrowserTest.RunsAudioVideoWebRTCCallInTwoTabsVP8 WebRtcDisableEncryptionFlagBrowserTest.VerifyEncryption
,
Oct 25
The first failing build seems to be this one: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Chromium%20OS%20ASan%20LSan%20Tests%20%281%29/29642 There are about 8 failures out of 100 builds (note that there are a bunch of viz_browser_tests failures that aren't the same thing).
,
Oct 25
Actually, a bunch of WebRtcBrowserTests are flaky on ASan LSan, viz or no: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=browser_tests&tests=WebRtcBrowserTest https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=viz_browser_tests&tests=WebRtcBrowserTest It's been going on since at least Oct 18, unfortunately the flakiness dashboard doesn't go further back than that.
,
Oct 25
Adding some content/browser/webrtc/OWNERS - can someone take a look at this? Otherwise I guess we'll just have to disable these tests on ASan.
,
Oct 25
I just noticed that the flaky WebRtcBrowserTests are the ones in chrome/browser/media/webrtc/, *not* the ones in content/browser/webrtc/, so adding better owners (at least there's some overlap :) ).
,
Oct 25
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/6d582d186fc5763adc328767d84a39d1b0e98376 commit 6d582d186fc5763adc328767d84a39d1b0e98376 Author: Marc Treib <treib@chromium.org> Date: Thu Oct 25 16:37:02 2018 Disable all WebRtcBrowserTests on ASan Many of these tests flakily leak and thus fail on ASan bots. TBR=phoglund@chromium.org Bug: 898546 Change-Id: Icd3cc5dc3a46d0194c807c27b66f7eb902b35794 Reviewed-on: https://chromium-review.googlesource.com/c/1299161 Reviewed-by: Marc Treib <treib@chromium.org> Commit-Queue: Marc Treib <treib@chromium.org> Cr-Commit-Position: refs/heads/master@{#602745} [modify] https://crrev.com/6d582d186fc5763adc328767d84a39d1b0e98376/chrome/browser/media/webrtc/webrtc_browsertest.cc
,
Oct 25
That's a pretty huge disable (all webrtc browser_tests on asan), assigning to tommi@ to ensure someone from the WebRTC team picks this up.
,
Oct 25
Dropping sheriff label since it's not a tree issue anymore though.
,
Oct 26
phoglund, hbos - This sounds like a sweeping change. Can we investigate the leaks? hbos - I'm assigning to you for now, feel free to reassign and/or break down into smaller tasks depending on how best to address the problem.
,
Oct 29
This ought to be due to a recent change. hbos@, let me know if I should get somebody to help you.
,
Oct 29
#11: Yes if you could please help, I have way too much stuff on my plate right now.
,
Oct 29
,
Oct 29
All right. Yves, can you help out here?
,
Oct 29
Thanks! :)
,
Oct 29
Looking into it! Is there a sensible change (libc++, compiler, ...) I should be aware of? :) Ah, I guess that's part of the investigation!
,
Oct 29
I was unable to find any link between WebRTC and the leaks (mostly related to the DOM, as far as I understand). I suspect these tests to be revealers, and webrtc not being the cause here. For instance, UkmBrowserTest.LogsTabId/0 is flaky as well, and triggers the same leaks. Cf https://chromium-swarm.appspot.com/task?id=40d49d96e7d5ca10&refresh=10&show_raw=1
,
Oct 29
The flakiness itself is interesting, and several reasons could explain it: 1. Race conditions. 2. Improper handling of errors (timeout, unavailable resource, ...). 3. You name it. To rule out 1/ it would be instructive to run ThreadSanitizer on the same set of tests. But according to the following dashboard, it doesn't seem to be the case (no tsan builder): https://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=WebRtcBrowserTest.RunsAudioVideoWebRTCCallInTwoTabsWithClonedCertificateEcdsa
,
Oct 30
Thanks Yves! Ok, so nothing of this is related to WebRTC. We could try explicitly closing all tabs we open in each test to work around this issue. I don't think we should be hunting down what these leaks are really about. So, idea: Call CloseWebContents(tab, false) from https://cs.chromium.org/chromium/src/chrome/browser/ui/browser_tabstrip.h?sq=package:chromium&g=0. I think we can just add a TearDown method that calls CloseWebContents on the left and right tab if they're != null: https://cs.chromium.org/chromium/src/chrome/browser/media/webrtc/webrtc_browsertest.cc?q=webrtc_browser&sq=package:chromium&g=0&l=46. Try doing that in a CL and re-enabling the test. Can you give that a shot Yves?
,
Nov 15
> So, idea: Call CloseWebContents(tab, false) Tabs are already closed via QuitBrowsers[1]. So calling CloseWebContents just involves dangling pointers :( [1] https://cs.chromium.org/chromium/src/chrome/test/base/in_process_browser_test.cc?rcl=1021636f0a71020c5b48cad17d8e65ecff71798c&l=592
,
Nov 15
What do you mean dangling pointers?
,
Nov 15
I was talking about left_tab_ and right_tab_ which points to freed memory. In short, the browser and its tabs are already being properly destructed.
,
Nov 15
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/8700ed14ef6030b990233f03b729c1b60f81bc43 commit 8700ed14ef6030b990233f03b729c1b60f81bc43 Author: Yves Gerey <yvesg@google.com> Date: Thu Nov 15 14:56:51 2018 Revert "Disable all WebRtcBrowserTests on ASan" This reverts commit 6d582d186fc5763adc328767d84a39d1b0e98376. Reason for revert: The leaks are not caused by WebRTC code, but merely revealed by the tests. We re-enable those tests: either the leaks have been fixed, or we'll add a suppression (less drastic than muting the whole WebRTC test suite). Original change's description: >Disable all WebRtcBrowserTests on ASan > >Many of these tests flakily leak and thus fail on ASan bots. > >TBR=phoglund@chromium.org > >Bug: 898546 >Change-Id: Icd3cc5dc3a46d0194c807c27b66f7eb902b35794 >Reviewed-on: https://chromium-review.googlesource.com/c/1299161 >Reviewed-by: Marc Treib <treib@chromium.org> >Commit-Queue: Marc Treib <treib@chromium.org> >Cr-Commit-Position: refs/heads/master@{#602745} Bug: chromium:898546 Change-Id: Idccff7827e6381e20060f7116cac75ee2b25018f Reviewed-on: https://chromium-review.googlesource.com/c/1337626 Reviewed-by: Patrik Höglund <phoglund@chromium.org> Commit-Queue: Yves Gerey <yvesg@google.com> Cr-Commit-Position: refs/heads/master@{#608366} [modify] https://crrev.com/8700ed14ef6030b990233f03b729c1b60f81bc43/chrome/browser/media/webrtc/webrtc_browsertest.cc
,
Nov 20
+ altimin,gab for input on DiscardableSharedMemoryManager and re-triage. > We re-enable those tests: either the leaks have been fixed, or we'll add a suppression (less drastic than muting the whole WebRTC test suite). The tests are still flaky, as witnessed by the dashboard [1]. I plan to add a suppression (ideally only applied for these WebRTC tests). Since it hides a real issue, this bug should be triaged so it isn't consigned to oblivion. The log has entries such as: [3101:3124:1118/202928.152114:WARNING:discardable_shared_memory_manager.cc(410)] Some MojoDiscardableSharedMemoryManagerImpls are still alive. They will be leaked. Since some stack traces also involve mojo, it looks like a suspect! [1] https://test-results.appspot.com/dashboards/flakiness_dashboard.html#testType=browser_tests&tests=WebRtcBrowserTest
,
Nov 20
Meta-issue: Leaks not reported. ------------------------------- Contrary to chromium-os builder [1], the regular linux lsan builder [2] is mostly green. For instance, build [3] is green. But we can witness leaks in a log [4] coming from this build. This happens because failing tests are retried. In my book, that a wrong way to deal with flakiness, since it amounts to mute new failures as well. Flaky tests should be fixed. In the meantime, existing sanitizer defects might be suppressed so that the bots remains sensitive to new defects. [1] https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Chromium%20OS%20ASan%20LSan%20Tests%20%281%29 [2] https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20ASan%20LSan%20Tests%20%281%29 [3] https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20ASan%20LSan%20Tests%20%281%29/52796 [4] https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8929328122055307232/+/steps/browser_tests/0/stdout
,
Nov 22
,
Nov 23
,
Nov 23
I'm landing another CL disabling all tests that are flaky on ASan builds: https://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=WebRtcBrowserTest yvesg@, to clarify something, this isn't fully disabling the tests but only disabling them when run in ASan bots. Given how flaky they are, having them run on ASan bots only has a negative effect as it's polluting the sheriff dashboard and we wouldn't very easily catch a real failure.
,
Nov 23
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e36cc582c8f2ed18935743a831ae46ffbe2f3b3f commit e36cc582c8f2ed18935743a831ae46ffbe2f3b3f Author: Mounir Lamouri <mlamouri@chromium.org> Date: Fri Nov 23 20:06:20 2018 Sheriff: Disable flaky WebRTC on ASan builds. Bug: 898546 Change-Id: I18f631b57d17eb9750ec37b1e7a00fa0d7d3cbea TBR: yvesg@chromium.org Reviewed-on: https://chromium-review.googlesource.com/c/1349450 Commit-Queue: Mounir Lamouri <mlamouri@chromium.org> Reviewed-by: Mounir Lamouri <mlamouri@chromium.org> Cr-Commit-Position: refs/heads/master@{#610660} [modify] https://crrev.com/e36cc582c8f2ed18935743a831ae46ffbe2f3b3f/chrome/browser/media/webrtc/webrtc_browsertest.cc
,
Dec 4
,
Dec 5
Lowering priority to 2 since the test is disabled. |
|||||||||||||||
►
Sign in to add a comment |
|||||||||||||||
Comment 1 by dalecur...@chromium.org
, Oct 24Components: Internals>Metrics>UKM Blink>Internals
Labels: Type-Bug
Status: Untriaged (was: Available)
Direct leak of 408 byte(s) in 3 object(s) allocated from: #0 0xffe763 in __interceptor_malloc /b/swarming/w/ir/kitchen-workdir/src/third_party/llvm/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3 #1 0xeab6230 in PartitionAllocGenericFlags base/allocator/partition_allocator/partition_alloc.h:354:48 #2 0xeab6230 in Alloc base/allocator/partition_allocator/partition_alloc.h:375 #3 0xeab6230 in WTF::Partitions::FastMalloc(unsigned long, char const*) third_party/blink/renderer/platform/wtf/allocator/partitions.h:114 #4 0x205930c1 in operator new third_party/blink/renderer/platform/wtf/ref_counted.h:44:3 #5 0x205930c1 in blink::SecurityOrigin::CreateUniqueOpaque() third_party/blink/renderer/platform/weborigin/security_origin.cc:193 #6 0x20592df3 in blink::SecurityOrigin::CreateWithReferenceOrigin(blink::KURL const&, blink::SecurityOrigin const*) third_party/blink/renderer/platform/weborigin/security_origin.cc:177:14 #7 0x2059397f in blink::SecurityOrigin::Create(blink::KURL const&) third_party/blink/renderer/platform/weborigin/security_origin.cc:188:10 #8 0x2129d73f in blink::Document::InitSecurityContext(blink::DocumentInit const&) third_party/blink/renderer/core/dom/document.cc:6387:23 #9 0x212991fe in blink::Document::Document(blink::DocumentInit const&, unsigned char) third_party/blink/renderer/core/dom/document.cc:740:3 #10 0x224e579f in blink::HTMLDocument::HTMLDocument(blink::DocumentInit const&, unsigned char) third_party/blink/renderer/core/html/html_document.cc:68:7 #11 0x224e5b73 in blink::HTMLDocument::Create(blink::DocumentInit const&) third_party/blink/renderer/core/html/html_document.cc:77:14 #12 0x213ae6e7 in blink::DOMImplementation::createDocument(WTF::String const&, blink::DocumentInit const&, bool) third_party/blink/renderer/core/dom/dom_implementation.cc:233:12 #13 0x21d93dd7 in blink::LocalDOMWindow::CreateDocument(WTF::String const&, blink::DocumentInit const&, bool) third_party/blink/renderer/core/frame/local_dom_window.cc:305:16 #14 0x21d9440b in blink::LocalDOMWindow::InstallNewDocument(WTF::String const&, blink::DocumentInit const&, bool) third_party/blink/renderer/core/frame/local_dom_window.cc:327:15 #15 0x23442924 in blink::DocumentLoader::InstallNewDocument(blink::KURL const&, blink::Document*, blink::WebGlobalObjectReusePolicy, WTF::AtomicString const&, WTF::AtomicString const&, blink::DocumentLoader::InstallNewDocumentReason, blink::ParserSynchronizationPolicy, blink::KURL const&) third_party/blink/renderer/core/loader/document_loader.cc:1161:45 #16 0x23441f06 in blink::DocumentLoader::CommitNavigation(WTF::AtomicString const&, blink::KURL const&) third_party/blink/renderer/core/loader/document_loader.cc:779:3 #17 0x2343da8d in blink::DocumentLoader::CommitData(char const*, unsigned long) third_party/blink/renderer/core/loader/document_loader.cc:795:3 #18 0x2343c2fa in blink::DocumentLoader::FinishedLoading(base::TimeTicks) third_party/blink/renderer/core/loader/document_loader.cc:525:7 #19 0x23444dc0 in blink::DocumentLoader::MaybeLoadEmpty() third_party/blink/renderer/core/loader/document_loader.cc:951:3 #20 0x2344518c in blink::DocumentLoader::StartLoading() third_party/blink/renderer/core/loader/document_loader.cc:961:7 #21 0x23473f34 in blink::FrameLoader::Init() third_party/blink/renderer/core/loader/frame_loader.cc:242:33 #22 0x21dd28af in blink::LocalFrame::Init() third_party/blink/renderer/core/frame/local_frame.cc:176:11 #23 0x23bc49cb in blink::SVGImage::DataChanged(bool) third_party/blink/renderer/core/svg/graphics/svg_image.cc:796:12 #24 0x203f4043 in blink::Image::SetData(scoped_refptr<blink::SharedBuffer>, bool) third_party/blink/renderer/platform/graphics/image.cc:115:10 #25 0x235118b5 in blink::ImageResourceContent::UpdateImage(scoped_refptr<blink::SharedBuffer>, blink::ResourceStatus, blink::ImageResourceContent::UpdateImageOption, bool, bool) third_party/blink/renderer/core/loader/resource/image_resource_content.cc:426:35 #26 0x23502979 in UpdateImage third_party/blink/renderer/core/loader/resource/image_resource.cc:735:31 #27 0x23502979 in blink::ImageResource::Finish(base::TimeTicks, base::SingleThreadTaskRunner*) third_party/blink/renderer/core/loader/resource/image_resource.cc:435 #28 0x10df0bec in blink::ResourceFetcher::ResourceForStaticData(blink::FetchParameters const&, blink::ResourceFactory const&, blink::SubstituteData const&) third_party/blink/renderer/platform/loader/fetch/resource_fetcher.cc:570:13 #29 0x10df6524 in blink::ResourceFetcher::RequestResource(blink::FetchParameters&, blink::ResourceFactory const&, blink::ResourceClient*, blink::SubstituteData const&) third_party/blink/renderer/platform/loader/fetch/resource_fetcher.cc:833:16 #30 0x234fc60e in blink::ImageResource::Fetch(blink::FetchParameters&, blink::ResourceFetcher*) third_party/blink/renderer/core/loader/resource/image_resource.cc:180:16 #31 0x2350c6b8 in blink::ImageResourceContent::Fetch(blink::FetchParameters&, blink::ResourceFetcher*) third_party/blink/renderer/core/loader/resource/image_resource_content.cc:124:29 #32 0x23b721e3 in blink::StyleFetchedImage::StyleFetchedImage(blink::Document const&, blink::FetchParameters&, bool) third_party/blink/renderer/core/style/style_fetched_image.cc:44:12 #33 0x20b5ff02 in Create third_party/blink/renderer/core/style/style_fetched_image.h:46:16 #34 0x20b5ff02 in blink::CSSImageValue::CacheImage(blink::Document const&, blink::FetchParameters::ImageRequestOptimization, blink::CrossOriginAttributeValue) third_party/blink/renderer/core/css/css_image_value.cc:81 Direct leak of 144 byte(s) in 2 object(s) allocated from: #0 0x102ace2 in operator new(unsigned long) /b/swarming/w/ir/kitchen-workdir/src/third_party/llvm/compiler-rt/lib/asan/asan_new_delete.cc:106:3 #1 0x16b62851 in make_unique<ukm::MojoUkmRecorder, mojo::InterfacePtr<ukm::mojom::UkmRecorderInterface> > buildtools/third_party/libc++/trunk/include/memory:3118:28 #2 0x16b62851 in ukm::MojoUkmRecorder::Create(service_manager::Connector*) services/metrics/public/cpp/mojo_ukm_recorder.cc:25 #3 0x212ca3d4 in blink::Document::UkmRecorder() third_party/blink/renderer/core/dom/document.cc:6312:7 #4 0x212c8790 in blink::Document::Shutdown() third_party/blink/renderer/core/dom/document.cc:2931:5 #5 0x21de48ee in blink::LocalFrame::ForceSynchronousDocumentInstall(WTF::AtomicString const&, scoped_refptr<blink::SharedBuffer>) third_party/blink/renderer/core/frame/local_frame.cc:1348:18 #6 0x23bc4b6e in blink::SVGImage::DataChanged(bool) third_party/blink/renderer/core/svg/graphics/svg_image.cc:812:10 #7 0x203f4043 in blink::Image::SetData(scoped_refptr<blink::SharedBuffer>, bool) third_party/blink/renderer/platform/graphics/image.cc:115:10 #8 0x235118b5 in blink::ImageResourceContent::UpdateImage(scoped_refptr<blink::SharedBuffer>, blink::ResourceStatus, blink::ImageResourceContent::UpdateImageOption, bool, bool) third_party/blink/renderer/core/loader/resource/image_resource_content.cc:426:35 #9 0x23502979 in UpdateImage third_party/blink/renderer/core/loader/resource/image_resource.cc:735:31 #10 0x23502979 in blink::ImageResource::Finish(base::TimeTicks, base::SingleThreadTaskRunner*) third_party/blink/renderer/core/loader/resource/image_resource.cc:435 #11 0x10df0bec in blink::ResourceFetcher::ResourceForStaticData(blink::FetchParameters const&, blink::ResourceFactory const&, blink::SubstituteData const&) third_party/blink/renderer/platform/loader/fetch/resource_fetcher.cc:570:13 #12 0x10df6524 in blink::ResourceFetcher::RequestResource(blink::FetchParameters&, blink::ResourceFactory const&, blink::ResourceClient*, blink::SubstituteData const&) third_party/blink/renderer/platform/loader/fetch/resource_fetcher.cc:833:16 #13 0x234fc60e in blink::ImageResource::Fetch(blink::FetchParameters&, blink::ResourceFetcher*) third_party/blink/renderer/core/loader/resource/image_resource.cc:180:16 #14 0x2350c6b8 in blink::ImageResourceContent::Fetch(blink::FetchParameters&, blink::ResourceFetcher*) third_party/blink/renderer/core/loader/resource/image_resource_content.cc:124:29 #15 0x23b721e3 in blink::StyleFetchedImage::StyleFetchedImage(blink::Document const&, blink::FetchParameters&, bool) third_party/blink/renderer/core/style/style_fetched_image.cc:44:12 #16 0x20b5ff02 in Create third_party/blink/renderer/core/style/style_fetched_image.h:46:16 #17 0x20b5ff02 in blink::CSSImageValue::CacheImage(blink::Document const&, blink::FetchParameters::ImageRequestOptimization, blink::CrossOriginAttributeValue) third_party/blink/renderer/core/css/css_image_value.cc:81 #18 0x2103f341 in blink::ElementStyleResources::LoadPendingImage(blink::ComputedStyle*, blink::StylePendingImage*, blink::FetchParameters::ImageRequestOptimization, blink::CrossOriginAttributeValue) third_party/blink/renderer/core/css/resolver/element_style_resources.cc:151:25 #19 0x21041340 in blink::ElementStyleResources::LoadPendingImages(blink::ComputedStyle*) third_party/blink/renderer/core/css/resolver/element_style_resources.cc:215:17 #20 0x21043161 in blink::ElementStyleResources::LoadPendingResources(blink::ComputedStyle*) third_party/blink/renderer/core/css/resolver/element_style_resources.cc:329:3 #21 0x210a5b54 in LoadPendingResources third_party/blink/renderer/core/css/resolver/style_resolver.cc:608:36 #22 0x210a5b54 in blink::StyleResolver::ApplyMatchedStandardProperties(blink::StyleResolverState&, blink::MatchResult const&, blink::StyleResolver::CacheSuccess const&, blink::StyleResolver::NeedsApplyPass&) third_party/blink/renderer/core/css/resolver/style_resolver.cc:1837 #23 0x21099275 in blink::StyleResolver::ApplyMatchedPropertiesAndCustomPropertyAnimations(blink::StyleResolverState&, blink::MatchResult const&, blink::Element const*) third_party/blink/renderer/core/css/resolver/style_resolver.cc:1597:5 #24 0x21097b1a in blink::StyleResolver::StyleForElement(blink::Element*, blink::ComputedStyle const*, blink::ComputedStyle const*, blink::RuleMatchingBehavior) third_party/blink/renderer/core/css/resolver/style_resolver.cc:776:5 #25 0x213efa1e in blink::Element::OriginalStyleForLayoutObject() third_party/blink/renderer/core/dom/element.cc:2355:46 #26 0x213eec29 in blink::Element::StyleForLayoutObject() third_party/blink/renderer/core/dom/element.cc:2324:46 #27 0x213f1e22 in blink::Element::RecalcOwnStyle(blink::StyleRecalcChange) third_party/blink/renderer/core/dom/element.cc:2519:17 #28 0x213f054a in blink::Element::RecalcStyle(blink::StyleRecalcChange) third_party/blink/renderer/core/dom/element.cc:2429:16 #29 0x2126bb16 in blink::ContainerNode::RecalcDescendantStyles(blink::StyleRecalcChange) third_party/blink/renderer/core/dom/container_node.cc:1404:18 #30 0x213f0894 in blink::Element::RecalcStyle(blink::StyleRecalcChange) third_party/blink/renderer/core/dom/element.cc:2466:7 #31 0x2126bb16 in blink::ContainerNode::RecalcDescendantStyles(blink::StyleRecalcChange) third_party/blink/renderer/core/dom/container_node.cc:1404:18 #32 0x213f0894 in blink::Element::RecalcStyle(blink::StyleRecalcChange) third_party/blink/renderer/core/dom/element.cc:2466:7 #33 0x2126bb16 in blink::ContainerNode::RecalcDescendantStyles(blink::StyleRecalcChange) third_party/blink/renderer/core/dom/container_node.cc:1404:18