How long has this been failing? This regression can be a very serious problem from a memory usage perspective as it means that the upper bound for discardable memory is no longer correctly maintained.

It's been happening at least since November 15th: https://uberchromegw.corp.google.com/i/internal.client.clank/builders/asan-clang-phone?numbuilds=200

Maybe even earlier, but the links to logs don't work before then, so it's hard to verify.

Discardable memory system was moved to //components from //content on Nov 2. This might be related to that.

penghuang@, can you take a look? This test verifies that memory is discarded properly by allocating more than 4GB. That should never cause a oom error.

I don't see why my CL can cause this problem.

mariakhomenko, Can I try a CL on this bot? How? I would like to create reverting CL and try it on this bot.

I found the bot has been failing on test DiscardableMemoryAddressSpace before my CL, but the stack is different.

https://luci-logdog.appspot.com/v/?s=chrome%2Fbb%2Finternal.client.clank%2Fasan-clang-phone%2F1227%2F%2B%2Frecipes%2Fsteps%2Fcontent_browsertests%2F0%2Fstdout

I 4271.384s run_tests_on_device(0622a7e6003b8c5b)  [ RUN      ] ChildThreadImplBrowserTest.DiscardableMemoryAddressSpace
I 4271.384s run_tests_on_device(0622a7e6003b8c5b)  [WARNING:dns_config_service_posix.cc(316)] Failed to read DnsConfig.
I 4271.384s run_tests_on_device(0622a7e6003b8c5b)  [ERROR:devtools_http_handler.cc(220)] Cannot start http server for devtools. Stop devtools.
I 4271.384s run_tests_on_device(0622a7e6003b8c5b)  [FATAL:gl_context.cc(216)] Check failed: error == GL_NO_ERROR || error == GL_CONTEXT_LOST_KHR. GL error was: 1285
I 4271.384s run_tests_on_device(0622a7e6003b8c5b)  #00 0x7a3edd67 /data/app-lib/org.chromium.content_browsertests_apk-1/libbase.cr.so+0x00141d67
I 4271.384s run_tests_on_device(0622a7e6003b8c5b)  #01 0x812af3e5 /data/app-lib/org.chromium.content_browsertests_apk-1/libgl_wrapper.cr.so+0x0010d3e5
I 4271.384s run_tests_on_device(0622a7e6003b8c5b)  #02 0x81d7407d /data/app-lib/org.chromium.content_browsertests_apk-1/libgpu.cr.so+0x000f107d
I 4271.384s run_tests_on_device(0622a7e6003b8c5b)  #03 0x81e36473 /data/app-lib/org.chromium.content_browsertests_apk-1/libgpu.cr.so+0x001b3473
I 4271.384s run_tests_on_device(0622a7e6003b8c5b)  #04 0x81dc7757 /data/app-lib/org.chromium.content_browsertests_apk-1/libgpu.cr.so+0x00144757
I 4271.384s run_tests_on_device(0622a7e6003b8c5b)  #05 0x81df07d9 /data/app-lib/org.chromium.content_browsertests_apk-1/libgpu.cr.so+0x0016d7d9
I 4271.384s run_tests_on_device(0622a7e6003b8c5b)  #06 0x81d37c01 /data/app-lib/org.chromium.content_browsertests_apk-1/libgpu.cr.so+0x000b4c01
I 4271.384s run_tests_on_device(0622a7e6003b8c5b)  #07 0x81d3a6fd /data/app-lib/org.chromium.content_browsertests_apk-1/libgpu.cr.so+0x000b76fd
I 4271.384s run_tests_on_device(0622a7e6003b8c5b)  #08 0x81feb0b9 /data/app-lib/org.chromium.content_browsertests_apk-1/libgpu.cr.so+0x003680b9
I 4271.384s run_tests_on_device(0622a7e6003b8c5b)  #09 0x81feabaf /data/app-lib/org.chromium.content_browsertests_apk-1/libgpu.cr.so+0x00367baf
I 4271.385s run_tests_on_device(0622a7e6003b8c5b)  #10 0x81fe70f9 /data/app-lib/org.chromium.content_browsertests_apk-1/libgpu.cr.so+0x003640f9
I 4271.385s run_tests_on_device(0622a7e6003b8c5b)  #11 0x81fd2987 /data/app-lib/org.chromium.content_browsertests_apk-1/libgpu.cr.so+0x0034f987
I 4271.385s run_tests_on_device(0622a7e6003b8c5b)  #12 0x81fd2665 /data/app-lib/org.chromium.content_browsertests_apk-1/libgpu.cr.so+0x0034f665
I 4271.385s run_tests_on_device(0622a7e6003b8c5b)  #13 0x7a3a03a9 /data/app-lib/org.chromium.content_browsertests_apk-1/libbase.cr.so+0x000f43a9
I 4271.385s run_tests_on_device(0622a7e6003b8c5b)  #14 0x7a39ff63 /data/app-lib/org.chromium.content_browsertests_apk-1/libbase.cr.so+0x000f3f63
I 4271.385s run_tests_on_device(0622a7e6003b8c5b)  #15 0x7a40b275 /data/app-lib/org.chromium.content_browsertests_apk-1/libbase.cr.so+0x0015f275
I 4271.385s run_tests_on_device(0622a7e6003b8c5b)  #16 0x7a40bb45 /data/app-lib/org.chromium.content_browsertests_apk-1/libbase.cr.so+0x0015fb45
I 4271.385s run_tests_on_device(0622a7e6003b8c5b)  #17 0x7a40c475 /data/app-lib/org.chromium.content_browsertests_apk-1/libbase.cr.so+0x00160475
I 4271.385s run_tests_on_device(0622a7e6003b8c5b)  #18 0x7a415533 /data/app-lib/org.chromium.content_browsertests_apk-1/libbase.cr.so+0x00169533
I 4271.385s run_tests_on_device(0622a7e6003b8c5b)  #19 0x7a40ac83 /data/app-lib/org.chromium.content_browsertests_apk-1/libbase.cr.so+0x0015ec83
I 4271.385s run_tests_on_device(0622a7e6003b8c5b)  #20 0x7a47995b /data/app-lib/org.chromium.content_browsertests_apk-1/libbase.cr.so+0x001cd95b
I 4271.385s run_tests_on_device(0622a7e6003b8c5b)  #21 0x7a4fde81 /data/app-lib/org.chromium.content_browsertests_apk-1/libbase.cr.so+0x00251e81
I 4271.385s run_tests_on_device(0622a7e6003b8c5b)  #22 0x7a4fe683 /data/app-lib/org.chromium.content_browsertests_apk-1/libbase.cr.so+0x00252683
I 4271.386s run_tests_on_device(0622a7e6003b8c5b)  #23 0x7a4e7fe5 /data/app-lib/org.chromium.content_browsertests_apk-1/libbase.cr.so+0x0023bfe5
I 4271.386s run_tests_on_device(0622a7e6003b8c5b)  #24 0x40624173 /system/lib/libc.so+0x0000d173
I 4271.386s run_tests_on_device(0622a7e6003b8c5b)  #25 0x4062430b /system/lib/libc.so+0x0000d30b
I 4271.386s run_tests_on_device(0622a7e6003b8c5b)  #26 0x74b80eff <unknown>
I 4271.386s run_tests_on_device(0622a7e6003b8c5b)  
I 4271.386s run_tests_on_device(0622a7e6003b8c5b)  [ERROR:test_suite.cc(270)] Currently running: ChildThreadImplBrowserTest.DiscardableMemoryAddressSpace
I 4271.386s run_tests_on_device(0622a7e6003b8c5b)  
I 4271.386s run_tests_on_device(0622a7e6003b8c5b)  ChildThreadImplBrowserTest.DiscardableMemoryAddressSpaceUNKNOWN
I 4271.386s run_tests_on_device(0622a7e6003b8c5b)  OK (0 tests)
I 4271.386s run_tests_on_device(0622a7e6003b8c5b)  Finished running tests on this device.
I 4271.386s run_tests_on_device(0694f579003c05c7)  Finished running tests on this device.

I don't think there's a trybot with this configuration. You could try to replicate locally, if you have an Android device.

+ Alex in case he has thoughts about the latter stack trace.

This test will try to allocate 4GB + 4MB discardable memory. And for the android device, max discardable memory limit is 128MB. Is it possible the device cannot handle 128MB memory requirement for an asan build chrome?

I tried add a printf in this test and run it locally and found out the upper bound for discardable memory is well maintained. So it is very likely the problem is because the device cannot handle 128MB memory requirement for the asan build chrome. So maybe we could reduce the limitation for asan android build or just disable this test for this bot.

WDYT?

https://codereview.chromium.org/2541573002/diff/1/content/child/child_thread_impl_browsertest.cc

Hmm, the device on that bot is Nexus 5, running KTU84P. So it has 2GB of RAM. It's possible, but surprising that it wouldn't be able to handle 128MB memory requirement.

I don't quite understand what this test is doing, but my preference would be to fix it for the asan bot, rather than disable it. If you can reduce the limitation and still have the test work, that sounds reasonable.

I reproduced this issue locally with Nexus 5 KTU84P.
I found several interesting things:
 1. The crash is because the content_browsertest runs out of memory address space. In the log, in the memory map, the last /dev/ashmem mapped address range is very close to the [stack] address range.
 2. The crash will happen when this test case allocates 391 discardable shared memory segments.
 3. I put some code in base::SharedMemory to track mmap and munmap. And found out mapped memory never over 260MB. It reaches 260MB quickly, but the mapped memory is keeping in the range 252MB to 260MB until crash.
 4. In Process memory map dump (See attached file):
   There are 66 mapped 4100KB /dev/ashmem blocks (I believe there are mapped by base::SharedMemory for discardable shared memory). I think half (33 blocks) is mapped by server side of discardable memory, and the other half (33 blocks) is mapped by client side of discardable memory. So I think the discardable shared memory does maintain the 128MB limitation.
   But there are many 2MB, 8MB and 11MB blocks are allocated between those /dev/ashmem blocks. They consumed a lot of memory address space. But not sure where are they come from. Any idea? (Maybe allocated by asan).

Deleted: log 332 KB

log 332 KB View Download

BTW, I modified discardable shared memory limit from 128MB to 64MB,  the test passed. Even if I increased the total allocated discardable shared memory in that test case from 4GB to 40GB, the test still passed.

So the problem is because asan debug build chrome consumes too much memory address space. I am pretty sure, it is not related to my refactor CL.

To fix the bot, we could ether disable the test for that bot, or reduce the discardable shared memory limit for asan debug build.

Seems like this test is not really testing the right thing in ASAN mode because it's dominated by ASAN bloat, so I'd suggest just disabling it conditionally for ASAN.

I agree we should disable conditionally for ASAN. I am not sure, but I think there's an ADDRESS_SANIZTIZER ifdef that you can use.

I created a CL which uses ADDRESS_SANIZTIZER and OS_ANDROID for the condition of this test.

https://codereview.chromium.org/2547063002/

Thanks for investigating!

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c1b2d7222c461133029f8dc7bee7c47145e442b1

commit c1b2d7222c461133029f8dc7bee7c47145e442b1
Author: penghuang <penghuang@chromium.org>
Date: Mon Dec 05 16:13:28 2016

Disable DiscardableMemoryAddressSpace for Android Asan build

BUG= 667837 

Review-Url: https://codereview.chromium.org/2547063002
Cr-Commit-Position: refs/heads/master@{#436307}

[modify] https://crrev.com/c1b2d7222c461133029f8dc7bee7c47145e442b1/content/renderer/render_thread_impl_browsertest.cc