Link to logs:

https://luci-logdog.appspot.com/v/?s=chromeos/buildbucket/cr-buildbucket.appspot.com/8935579443772288560/+/steps/UnitTest/0/stdout

It doesn't look like it's reporting a leak.

It didn't fail in this run at 2018-09-10 08:09: http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8935760638046188128

I think that the first failure was in this build that started 2018-09-11 08:02: http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8935670041274521488

And then the most recent build at 2018-09-12 08:09 also failed: http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8935579443772288560

libchromeos-ui's code hasn't had substantial changes in a very long time. It was converted to build with gn rather than GYP in https://crrev.com/c/1201282, which went in on 2018-09-09, but I think that should've been in the passing build described above, so it's unlikely to be responsible.

The stack trace looks more like a gtest issue than a libchromeos-ui issue. chromium_command_builder_unittest.cc:130 is an invocation of the TEST_F macro:

TEST_F(ChromiumCommandBuilderTest, IsTestBuild) {

Bluetooth and drivefs are also failing, are they different errors?
Those are not migrated to GN yet.

For chromeos-ui, I double-checked compiler options by GN and GYP, but coundn't find any essential difference.

I think that someone on the toolchain team needs to take a look at this. I don't see any evidence in the stack traces that this is related to libchromeos-ui's code. It looks like the error is caused by a gtest macro, but there aren't any recent updates to gtest.

Manoj, I'm assigning this to you since you made an earlier fix to gtest last year. Please let me know if I'm missing something.

Pretty sure that it was caused by gyp to gn migration.

https://chromium-review.googlesource.com/c/chromiumos/platform2/+/1201282 landed on Sep 9, 10:34 am
betty-asan run on Sept 9 (started at 8am before the CL landed): Passed
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?id=2925928 

betty-asan run on Sept 10 : Failed
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?id=2928295

Looking at amd64-generic-asan:
Run that started at 9am before the CL landed: Passed
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8935846625728386016

Next run that started at 12pm and included this CL: Failed
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8935837111261633200

oka@ Can you inspect the command lines differences between gyp vs gn builds? In particular, pay attention to the "-fvisibility=" and "-fsanitize=" flags.

Note that to repo:
$ ./setup_board --board=amd64-generic --profile=asan
(Slightly reduced list of packages for shorter build times)
$ ./build_packages --board=amd64-generic virtual/implicit-system libchromeos-ui
(Run with tests enabled)
$ FEATURES=test emerge-amd64-generic libchromeos-ui

Ah sorry.
Recent changes about asan was not reflected to GN configurations file.
That's why -fsanitize flag is not passed to libchromeos-ui .
I am going to submit https://chromium-review.googlesource.com/c/chromiumos/platform2/+/1223930.

It has been merged.

The unit test is still failing. oka@ Can you capture the compiler command lines for gyp and gn builds and compare them?

vpn-manager has started failing with same error in unit test:
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8935137784506876400

Hmm #6 passes on local environment. Not sure what makes difference.

Here is the GN compiler flags and GYP compiler flags.
I will examine them tomorrow.

If it's important to fix the test, please feel free to revert the CL.

Deleted: gyp_compiler_options.txt 9.8 KB

gyp_compiler_options.txt 9.8 KB View Download

Deleted: gn_compiler_options.txt 9.8 KB

gn_compiler_options.txt 9.8 KB View Download

This is the first failure in amd64-generic-asan:

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?id=2926357

Bad version:  R71-11051.0.0-b2926357
Good version: R71-11050.0.0-b2925965
Delta: https://crosland.corp.google.com/log/11050.0.0..11051.0.0

Only change mentioning libchromeos-ui:

	879f680c	1201282	767517	Fri Aug 31 04:24:16 2018	oka@chromium.org	libchromeos-ui: migrate the package to GN

Which basically confirms what has already been identified.

I'd hate to have to revert back to using GYP, especially if we have the same root cause for the vpn-manager failures (that's next on my list to triage).

We can definitely wait until tomorrow, thanks for investigating!

First failure for vpn-manager:
https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?id=2943276

Bad version: R71-11067.0.0-b2943276
Good version: R71-11066.0.0-b2942607
Delta: https://crosland.corp.google.com/log/11066.0.0..11067.0.0

Suspect CL is the same root cause as suspected:

	2b8a5484	1201248	767517	Fri Aug 31 04:24:16 2018	oka@chromium.org	vpn: migrate the package to GN

The command lines in #11 are identical so I am not sure what could be causing the fails.

oka@ can you dump and compare output of env before the test starts running? Maybe, something is different here which is causing the fails.

Sorry I couldn't figure it out today. Thank you for creating the reverts. Let's submit them.

OK. I reproduced the error locally using betty.
Here are the compile commands of GYP and GN.

Commands I ran are as follows:

% cros_sdk BOARD=betty
$ FEATURES=test emerge-amd64-generic libchromeos-ui
$ ./setup_board --board=$BOARD --profile=asan
$ ./build_packages --board=$BOARD libchromeos-ui
$ VERBOSE=1 FEATURES=test emerge-$BOARD libchromeos-ui

Are the attached logs in #18 correct? The command lines are referring to amd64-generic, not betty.

I'm sorry. They are not correct. Re-uploaded correct files.

Deleted: gyp_compiler_options.txt 9.3 KB

gyp_compiler_options.txt 9.3 KB View Download

Deleted: gn_compiler_options.txt 7.6 KB

gn_compiler_options.txt 7.6 KB View Download

One possible suspect:
GYP linked binaries have the linker options that are missing from GN builds:
-Wl,-rpath=$ORIGIN/lib/
-Wl,-rpath-link=lib/

There is also another giant if-else statement in GYP builds that is not present in GN builds, but not sure how important that is:
mmon-mk/libtestrunner.a obj/common-mk/testrunner.testrunner.o
[6/8] if [ ! -e lib/libchromeos-ui-395517.so -o ! -e
lib/libchromeos-ui-395517.so.TOC ]; then x86_64-cros-linux-gnu-clang++ -shared
-Wl,-O1 -Wl,-O2 -Wl,--as-needed -fsanitize=address -fsanitize=alignment
-fsanitize=shift -Wl,-z,relro -Wl,-z,noexecstack -Wl,-z,now -Wl,--as-needed
--sysroot=/build/betty -o lib/libchromeos-ui-395517.so
-Wl,-soname=libchromeos-ui-395517.so @lib/libchromeos-ui-395517.so.rsp && {
readelf -d lib/libchromeos-ui-395517.so | grep SONAME ; nm -gD -f p
lib/libchromeos-ui-395517.so | cut -f1-2 -d' '; } >
lib/libchromeos-ui-395517.so.TOC; else x86_64-cros-linux-gnu-clang++ -shared
-Wl,-O1 -Wl,-O2 -Wl,--as-needed -fsanitize=address -fsanitize=alignment
-fsanitize=shift -Wl,-z,relro -Wl,-z,noexecstack -Wl,-z,now -Wl,--as-needed
--sysroot=/build/betty -o lib/libchromeos-ui-395517.so
-Wl,-soname=libchromeos-ui-395517.so @lib/libchromeos-ui-395517.so.rsp && {
readelf -d lib/libchromeos-ui-395517.so | grep SONAME ; nm -gD -f p
lib/libchromeos-ui-395517.so | cut -f1-2 -d' '; } >
lib/libchromeos-ui-395517.so.tmp && if ! cmp -s lib/libchromeos-ui-395517.so.tmp
lib/libchromeos-ui-395517.so.TOC; then mv lib/libchromeos-ui-395517.so.tmp
lib/libchromeos-ui-395517.so.TOC ; fi; fi

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/b72f49940c39e5214be9a2f987aaf04ebd4503ba

commit b72f49940c39e5214be9a2f987aaf04ebd4503ba
Author: Manoj Gupta <manojgupta@google.com>
Date: Tue Sep 18 21:04:22 2018

Revert GN migration for libchrome-ui/vpn-manager.

Revert CL:1201248 and CL:1201282 since they are breaking asan
builder.

BUG= chromium:883512 
TEST=FEATURES=test USE=asan emerge-amd64-generic passes.
TEST=https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8935100673138787696

Change-Id: Ieab9432f8fdc1b8b3b1a5721bfe20b6ec322891d
Reviewed-on: https://chromium-review.googlesource.com/1228601
Commit-Ready: Manoj Gupta <manojgupta@chromium.org>
Tested-by: Manoj Gupta <manojgupta@chromium.org>
Reviewed-by: Keigo Oka <oka@chromium.org>

[add] https://crrev.com/b72f49940c39e5214be9a2f987aaf04ebd4503ba/vpn-manager/vpn-manager.gyp
[delete] https://crrev.com/991fdf38e7b067401e10e962ab7258acae0c4a37/vpn-manager/BUILD.gn
[add] https://crrev.com/b72f49940c39e5214be9a2f987aaf04ebd4503ba/libchromeos-ui/libchromeos-ui.gyp
[delete] https://crrev.com/991fdf38e7b067401e10e962ab7258acae0c4a37/libchromeos-ui/BUILD.gn

Hmm. Ok. It turned out to be the matter of library ordering.

On compiling libchromeos-ui-test, if -lgtest is before -lbase-395517 (gyp case), heap buffer overflow doesn't happen. However, if -lgtest is after -lbase-395517 (gn case), it does happen.

So if I make sure -lgtest comes first tweaking BUILD.gn, the issue is actually gone, though it doesn't seem a correct solution.

Interesting, seems like some symbols in libbase are being exported when they shouldn't be since gtest and libbase symbol set should be exclusive. 

Let's fix the library ordering so that this issue doesn't block GN migration.

Filed crbug.com/887845 to fix the root cause.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/c322077a91fb193bb0f75b7b73d78b131dc580d9

commit c322077a91fb193bb0f75b7b73d78b131dc580d9
Author: Keigo Oka <oka@chromium.org>
Date: Fri Sep 21 13:58:44 2018

Correct library ordering to fix heap buffer overflow

BUG= chromium:883512 
BUG=chromium:767517
TEST=precq passes

Change-Id: I7235242ad9e85c7e1c6887eef6091e152aa57430
Reviewed-on: https://chromium-review.googlesource.com/1234697
Commit-Ready: Keigo Oka <oka@chromium.org>
Tested-by: Keigo Oka <oka@chromium.org>
Reviewed-by: Keigo Oka <oka@chromium.org>

[modify] https://crrev.com/c322077a91fb193bb0f75b7b73d78b131dc580d9/common-mk/testrunner/BUILD.gn
[modify] https://crrev.com/c322077a91fb193bb0f75b7b73d78b131dc580d9/goldfishd/BUILD.gn
[modify] https://crrev.com/c322077a91fb193bb0f75b7b73d78b131dc580d9/libpasswordprovider/BUILD.gn
[modify] https://crrev.com/c322077a91fb193bb0f75b7b73d78b131dc580d9/common-mk/BUILD.gn
[modify] https://crrev.com/c322077a91fb193bb0f75b7b73d78b131dc580d9/touch_keyboard/BUILD.gn
[modify] https://crrev.com/c322077a91fb193bb0f75b7b73d78b131dc580d9/hermes/BUILD.gn
[modify] https://crrev.com/c322077a91fb193bb0f75b7b73d78b131dc580d9/hammerd/BUILD.gn
[modify] https://crrev.com/c322077a91fb193bb0f75b7b73d78b131dc580d9/libcontainer/BUILD.gn

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/b9d7e317d1cbeba858ecce0a9f47eab6a6c00260

commit b9d7e317d1cbeba858ecce0a9f47eab6a6c00260
Author: Keigo Oka <oka@chromium.org>
Date: Mon Sep 24 13:49:58 2018

libchromeos-ui: reland GN migration fixing ASAN failure

Fixed the issue moving //common-mk:test at the top of configs.

BUG= chromium:883512 
BUG=chromium:767517
TEST=Following commands doesn't produce ASAN error. Though the test itself
fails samely as GYP build on the same assert.
    % cros_sdk BOARD=betty
    $ ./setup_board --board=$BOARD --profile=asan
    $ ./build_packages --board=$BOARD libchromeos-ui
    $ VERBOSE=1 FEATURES=test emerge-$BOARD libchromeos-ui

Change-Id: I94cecf3e086eb26a6bd276de0381e23a85606c41
Reviewed-on: https://chromium-review.googlesource.com/1234094
Commit-Ready: Keigo Oka <oka@chromium.org>
Tested-by: Keigo Oka <oka@chromium.org>
Reviewed-by: Keigo Oka <oka@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[delete] https://crrev.com/338cca8ec776ca69c51294b45af8cf99c416497e/libchromeos-ui/libchromeos-ui.gyp
[add] https://crrev.com/b9d7e317d1cbeba858ecce0a9f47eab6a6c00260/libchromeos-ui/BUILD.gn

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform2/+/65a255d1ec9dbfde65adb8b3c6a200c2ff1a6ecd

commit 65a255d1ec9dbfde65adb8b3c6a200c2ff1a6ecd
Author: Keigo Oka <oka@chromium.org>
Date: Mon Sep 24 13:49:55 2018

vpn: reland GN migration with fixing ASAN failure

Fixed the issue moving //common-mk:test at the top of configs.

BUG= chromium:883512 
BUG=chromium:767517
TEST=% cros_sdk BOARD=betty
    $ ./setup_board --board=$BOARD --profile=asan
    $ ./build_packages --board=$BOARD vpn-manager
    $ VERBOSE=1 FEATURES=test emerge-$BOARD vpn-manager

Change-Id: I65f500746e87e37d69ce98b0505860dd80be8de7
Reviewed-on: https://chromium-review.googlesource.com/1234095
Commit-Ready: Keigo Oka <oka@chromium.org>
Tested-by: Keigo Oka <oka@chromium.org>
Reviewed-by: Keigo Oka <oka@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[delete] https://crrev.com/690b750f755f7282ec37daa993e894d08d656b6d/vpn-manager/vpn-manager.gyp
[add] https://crrev.com/65a255d1ec9dbfde65adb8b3c6a200c2ff1a6ecd/vpn-manager/BUILD.gn