New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 807685 link

Starred by 1 user

Issue metadata

Status: Verified
Owner:
Closed: Feb 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug
Build-Toolchain



Sign in to add a comment

chromeos-config-tools failing in asan

Project Member Reported by newcomer@chromium.org, Jan 31 2018

Issue description

chromeos-config-tools-0.0.2-r949:  * ASAN error detected:
chromeos-config-tools-0.0.2-r949:  * =================================================================
chromeos-config-tools-0.0.2-r949:  * ==23==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x7fbcd2a6a010 in thread T0
chromeos-config-tools-0.0.2-r949:  *     #0 0x7fbcd1338a32 in operator delete(void*) ??:0:0
chromeos-config-tools-0.0.2-r949:  *     #1 0x7fbcd0f30c4b in base::CommandLine::InitFromArgv(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) ??:0:0
chromeos-config-tools-0.0.2-r949:  * 
chromeos-config-tools-0.0.2-r949:  * Address 0x7fbcd2a6a010 is a wild pointer.
chromeos-config-tools-0.0.2-r949:  * SUMMARY: AddressSanitizer: bad-free (/var/cache/portage/chromeos-base/chromeos-config-tools/out/Default/cros_config+0xf0a32)
chromeos-config-tools-0.0.2-r949:  * ==23==ABORTING
chromeos-config-tools-0.0.2-r949: 
chromeos-config-tools-0.0.2-r949: 
chromeos-config-tools-0.0.2-r949:  * ASAN error detected:
chromeos-config-tools-0.0.2-r949:  * =================================================================
chromeos-config-tools-0.0.2-r949:  * ==24==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x7f02944cb010 in thread T0
chromeos-config-tools-0.0.2-r949:  *     #0 0x7f0292a45a32 in operator delete(void*) ??:0:0
chromeos-config-tools-0.0.2-r949:  *     #1 0x7f029263dc4b in base::CommandLine::InitFromArgv(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) ??:0:0
chromeos-config-tools-0.0.2-r949:  * 
chromeos-config-tools-0.0.2-r949:  * Address 0x7f02944cb010 is a wild pointer.
chromeos-config-tools-0.0.2-r949:  * SUMMARY: AddressSanitizer: bad-free (/var/cache/portage/chromeos-base/chromeos-config-tools/out/Default/cros_config+0xf0a32)
chromeos-config-tools-0.0.2-r949:  * ==24==ABORTING
chromeos-config-tools-0.0.2-r949: 
chromeos-config-tools-0.0.2-r949: 
chromeos-config-tools-0.0.2-r949:  * ASAN error detected:
chromeos-config-tools-0.0.2-r949:  * =================================================================
chromeos-config-tools-0.0.2-r949:  * ==25==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x7f1ea1ff8010 in thread T0
chromeos-config-tools-0.0.2-r949:  *     #0 0x7f1ea01b6a32 in operator delete(void*) ??:0:0
chromeos-config-tools-0.0.2-r949:  *     #1 0x7f1e9fdaec4b in base::CommandLine::InitFromArgv(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) ??:0:0
chromeos-config-tools-0.0.2-r949:  * 
chromeos-config-tools-0.0.2-r949:  * Address 0x7f1ea1ff8010 is a wild pointer.
chromeos-config-tools-0.0.2-r949:  * SUMMARY: AddressSanitizer: bad-free (/var/cache/portage/chromeos-base/chromeos-config-tools/out/Default/cros_config+0xf0a32)
chromeos-config-tools-0.0.2-r949:  * ==25==ABORTING
chromeos-config-tools-0.0.2-r949: 
chromeos-config-tools-0.0.2-r949: 
chromeos-config-tools-0.0.2-r949:  * ASAN error detected:
chromeos-config-tools-0.0.2-r949:  * =================================================================
chromeos-config-tools-0.0.2-r949:  * ==26==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x7f6c1a628010 in thread T0
chromeos-config-tools-0.0.2-r949:  *     #0 0x7f6c18d77a32 in operator delete(void*) ??:0:0
chromeos-config-tools-0.0.2-r949:  *     #1 0x7f6c1896fc4b in base::CommandLine::InitFromArgv(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) ??:0:0
chromeos-config-tools-0.0.2-r949:  * 
chromeos-config-tools-0.0.2-r949:  * Address 0x7f6c1a628010 is a wild pointer.
chromeos-config-tools-0.0.2-r949:  * SUMMARY: AddressSanitizer: bad-free (/var/cache/portage/chromeos-base/chromeos-config-tools/out/Default/cros_config+0xf0a32)
chromeos-config-tools-0.0.2-r949:  * ==26==ABORTING
chromeos-config-tools-0.0.2-r949: 
chromeos-config-tools-0.0.2-r949: 
chromeos-config-tools-0.0.2-r949:  * ASAN error detected:
chromeos-config-tools-0.0.2-r949:  * =================================================================
chromeos-config-tools-0.0.2-r949:  * ==27==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x7fe8e032c010 in thread T0
chromeos-config-tools-0.0.2-r949:  *     #0 0x7fe8dea66a32 in operator delete(void*) ??:0:0
chromeos-config-tools-0.0.2-r949:  *     #1 0x7fe8de65ec4b in base::CommandLine::InitFromArgv(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) ??:0:0
chromeos-config-tools-0.0.2-r949:  * 
chromeos-config-tools-0.0.2-r949:  * Address 0x7fe8e032c010 is a wild pointer.
chromeos-config-tools-0.0.2-r949:  * SUMMARY: AddressSanitizer: bad-free (/var/cache/portage/chromeos-base/chromeos-config-tools/out/Default/cros_config+0xf0a32)
chromeos-config-tools-0.0.2-r949:  * ==27==ABORTING
chromeos-config-tools-0.0.2-r949: 

and many more.
 
Cc: euge...@chromium.org
Components: Tools>ChromeOS-Toolchain
Labels: libcxx
Evgenii, 

Any idea about this fail? How do I avoid asan complains here.

~FilePath() itself is empty so it will be calling std::string default destructor and that should not cause these bad pointer errors.

void CommandLine::InitFromArgv(const StringVector& argv) {
  argv_ = StringVector(1);
  switches_.clear();
  switches_by_stringpiece_.clear();
  begin_args_ = 1;
  SetProgram(argv.empty() ? FilePath() : FilePath(argv[0])); <- complaining here.
  AppendSwitchesAndArguments(this, argv);
}

This looks really strange. Do you know why is there no debug info anywhere, and why the stack traces are truncated?

Could you track where did this pointer come from? It does not look like something that ASan malloc()/operator new() would return.
These are the logs from my local build. 

* ASAN error detected:
 * =================================================================
 * ==23==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x55bd0193a010 in thread T0
 *     #0 0x55bd008b1a32 in operator delete(void*) ??:0:0
 *     #1 0x7fac89dc7c5b in base::CommandLine::InitFromArgv(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /build/amd64-generic/tmp/portage/chromeos-base/libchrome-395517-r14/work/libchrome-395517/base/command_line.cc:258:3
 * 
 * Address 0x55bd0193a010 is a wild pointer.
 * SUMMARY: AddressSanitizer: bad-free (/var/cache/portage/chromeos-base/chromeos-config-tools/out/Default/cros_config+0xf0a32)
 * ==23==ABORTING


 * ASAN error detected:
 * =================================================================
 * ==24==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x5582ef5ca010 in thread T0
 *     #0 0x5582ed41ea32 in operator delete(void*) ??:0:0
 *     #1 0x7f4b258d8c5b in base::CommandLine::InitFromArgv(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /build/amd64-generic/tmp/portage/chromeos-base/libchrome-395517-r14/work/libchrome-395517/base/command_line.cc:258:3
 * 
 * Address 0x5582ef5ca010 is a wild pointer.
 * SUMMARY: AddressSanitizer: bad-free (/var/cache/portage/chromeos-base/chromeos-config-tools/out/Default/cros_config+0xf0a32)
 * ==24==ABORTING


 * ASAN error detected:
 * =================================================================
 * ==25==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x561b36b2e010 in thread T0
 *     #0 0x561b345eda32 in operator delete(void*) ??:0:0
 *     #1 0x7f0d181f7c5b in base::CommandLine::InitFromArgv(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /build/amd64-generic/tmp/portage/chromeos-base/libchrome-395517-r14/work/libchrome-395517/base/command_line.cc:258:3
 * 
 * Address 0x561b36b2e010 is a wild pointer.
 * SUMMARY: AddressSanitizer: bad-free (/var/cache/portage/chromeos-base/chromeos-config-tools/out/Default/cros_config+0xf0a32)
 * ==25==ABORTING


 * ASAN error detected:
 * =================================================================
 * ==26==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x5563b535b010 in thread T0
 *     #0 0x5563b2a14a32 in operator delete(void*) ??:0:0
 *     #1 0x7f9ed50b5c5b in base::CommandLine::InitFromArgv(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /build/amd64-generic/tmp/portage/chromeos-base/libchrome-395517-r14/work/libchrome-395517/base/command_line.cc:258:3
 * 
 * Address 0x5563b535b010 is a wild pointer.
 * SUMMARY: AddressSanitizer: bad-free (/var/cache/portage/chromeos-base/chromeos-config-tools/out/Default/cros_config+0xf0a32)
 * ==26==ABORTING


 * ASAN error detected:
 * =================================================================
 * ==27==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x559b5eba8010 in thread T0
 *     #0 0x559b5d345a32 in operator delete(void*) ??:0:0
 *     #1 0x7fbdd91e3c5b in base::CommandLine::InitFromArgv(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /build/amd64-generic/tmp/portage/chromeos-base/libchrome-395517-r14/work/libchrome-395517/base/command_line.cc:258:3
 * 
 * Address 0x559b5eba8010 is a wild pointer.
 * SUMMARY: AddressSanitizer: bad-free (/var/cache/portage/chromeos-base/chromeos-config-tools/out/Default/cros_config+0xf0a32)
 * ==27==ABORTING


 * ASAN error detected:
 * =================================================================
 * ==28==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x55e45d713010 in thread T0
 *     #0 0x55e45b31da32 in operator delete(void*) ??:0:0
 *     #1 0x7f35ded9fc5b in base::CommandLine::InitFromArgv(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /build/amd64-generic/tmp/portage/chromeos-base/libchrome-395517-r14/work/libchrome-395517/base/command_line.cc:258:3
 * 
 * Address 0x55e45d713010 is a wild pointer.
 * SUMMARY: AddressSanitizer: bad-free (/var/cache/portage/chromeos-base/chromeos-config-tools/out/Default/cros_config+0xf0a32)
 * ==28==ABORTING


 * ASAN error detected:
 * =================================================================
 * ==29==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x55753b246010 in thread T0
 *     #0 0x557538c29a32 in operator delete(void*) ??:0:0
 *     #1 0x7fcb4640fc5b in base::CommandLine::InitFromArgv(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /build/amd64-generic/tmp/portage/chromeos-base/libchrome-395517-r14/work/libchrome-395517/base/command_line.cc:258:3
 * 
 * Address 0x55753b246010 is a wild pointer.
 * SUMMARY: AddressSanitizer: bad-free (/var/cache/portage/chromeos-base/chromeos-config-tools/out/Default/cros_config+0xf0a32)
 * ==29==ABORTING


 * ASAN error detected:
 * =================================================================
 * ==30==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x555d4ccbd010 in thread T0
 *     #0 0x555d4a235a32 in operator delete(void*) ??:0:0
 *     #1 0x7fa45507bc5b in base::CommandLine::InitFromArgv(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) /build/amd64-generic/tmp/portage/chromeos-base/libchrome-395517-r14/work/libchrome-395517/base/command_line.cc:258:3
 * 
 * Address 0x555d4ccbd010 is a wild pointer.
 * SUMMARY: AddressSanitizer: bad-free (/var/cache/portage/chromeos-base/chromeos-config-tools/out/Default/cros_config+0xf0a32)
 * ==30==ABORTING

 * The complete build log is located at '/build/amd64-generic/tmp/portage/logs/chromeos-base:chromeos-config-tools-0.0.2-r944:20180201-041547.log'.
 * For convenience, a symlink to the build log is located at '/build/amd64-generic/tmp/portage/chromeos-base/chromeos-config-tools-0.0.2-r944/temp/build.log'.
 * The ebuild environment file is located at '/build/amd64-generic/tmp/portage/chromeos-base/chromeos-config-tools-0.0.2-r944/temp/environment'.
 * Working directory: '/mnt/host/source/src/platform2/chromeos-config'
 * S: '/mnt/host/source/src/platform2/chromeos-config'
All I can say is ASan allocator got a pointer in free/delete that it never produced. Something is wrong in the way this binary was linked. Is there a second allocator somewhere?
Cc: derat@chromium.org
I see a custom allocator in libbase (used as libchrome in Chrome OS) https://cs.chromium.org/chromium/src/base/allocator/allocator_shim_override_cpp_symbols.h?l=16

I don't know if that allocator is being used by libchrome.

Comment 7 by derat@chromium.org, Feb 1 2018

Cc: primiano@chromium.org sjg@chromium.org wfh@chromium.org
I don't know anything about //base/allocator, but it looks like it may live behind USE_EXPERIMENTAL_ALLOCATOR_SHIM, which I don't think we're explicitly setting anywhere within Chrome OS.
Cc: vapier@chromium.org
The tests passed after I commented out the malloc/new overloads from //base/allocator.

vapier@ How do you suggest proceeding from here? I can create a patch to disable these overloads when USE="asan". But maybe we should not be compiling them in first place?
The allocator shim (which is not experimental anymore) is disabled under asan (and any sanitizer). What are your gn flags? There i something wrong in your build config at some point if that is true , the allocator shim is supposed to be disabled under sanitizers.
But to be honest I don't see how this is related to the shim. I don't see that in the backtrace you attached. 
What I see is that you are deleting a malloc-ed pointer with delete. 
Very likely you have a unique_ptr or vector<char> which is fed with a strdup or similar. You need a base::freedeleter somewhere 
mail@ libchrome is built with scons in ChromeOS, not GN/ninja.

https://chromium.googlesource.com/aosp/platform/external/libchrome/+/master/SConstruct

Maybe these files should not be included in Chrome OS builds (or patched out when asan is used):

                allocator/allocator_extension.cc
                allocator/allocator_shim.cc
                allocator/allocator_shim_default_dispatch_to_glibc.cc
I have no idea about cros  workflows but, as a base/allocator owner,  I warmly suggest to stick to what chrome does under gn.
Just look at the allocator/build.gn and do the same. It's a combination of not including files + generating the right build flag file (allocator/features.h)

Still I am not convinced here that your problem is base::allocator. 
Can you point out which line in the stack trace makes you believe so? 
The problem is: ASan caught the delete but new went through allocator's new instead of ASan's one. 
So ASan complained when its delete was called to delete a pointer that didn't pass through it
BTW, Thanks Evgenii for spending time in debugging this with me.
Cc: manojgupta@chromium.org
 Issue 807684  has been merged into this issue.
 Issue 807681  has been merged into this issue.
Primiano, Here is the information about libchrome in CrOS builds https://www.chromium.org/chromium-os/packages/libchrome .
Since there is pretty heavy customization involved and given my limited (or rather none) experience in using libchrome, I believe moving libchrome builds to GN is beyond the scope of this bug.
Re #16: I am not suggesting to switch to GN. I am just suggesting to make sure that the build flags used by CrOS in scons match what happens in official chrome builds.

> The problem is: ASan caught the delete but new went through allocator's new instead of ASan's one. 
> So ASan complained when its delete was called to delete a pointer that didn't pass through it
Yup I see this, but also if the allocator shim was involved you would have seen ShimFree() in the call stack.

Check that you are not in an inconsistent state where the shim is disabled, but its operator new/delete overrides are still in place.
Specifically check that neither USE_ALLOCATOR_SHIM is defined, nor allocator_shim.cc is built when using ASan.
Project Member

Comment 18 by bugdroid1@chromium.org, Feb 6 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/a5ce0762807d30f3d511f95c81d15870c8eac73b

commit a5ce0762807d30f3d511f95c81d15870c8eac73b
Author: Manoj Gupta <manojgupta@google.com>
Date: Tue Feb 06 12:55:41 2018

libchrome: Disable memory allocator in asan builds.

Memory allocator in //base/allocator should not be used
with address sanitizer. Otherwise, it breaks asan builds since
many memory allocations will go to the base allocator instead
of asan.

BUG= chromium:807685 
TEST=No more asan complains about delete calls.

Change-Id: Ic81d59d2c293194c2d229659025605fafa8168b8
Reviewed-on: https://chromium-review.googlesource.com/900684
Commit-Ready: Manoj Gupta <manojgupta@chromium.org>
Tested-by: Manoj Gupta <manojgupta@chromium.org>
Reviewed-by: Luis Lozano <llozano@chromium.org>

[rename] https://crrev.com/a5ce0762807d30f3d511f95c81d15870c8eac73b/chromeos-base/libchrome/libchrome-395517-r16.ebuild
[add] https://crrev.com/a5ce0762807d30f3d511f95c81d15870c8eac73b/chromeos-base/libchrome/files/libchrome-395517-Disable-memory-allocator.patch
[modify] https://crrev.com/a5ce0762807d30f3d511f95c81d15870c8eac73b/chromeos-base/libchrome/libchrome-395517.ebuild

Status: Verified (was: Untriaged)
All asan unit tests passed at https://build.chromium.org/p/chromiumos/builders/amd64-generic-asan/builds/23262
Labels: libcxx_asan

Sign in to add a comment