New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 638968 link

Starred by 2 users

Issue metadata

Status: Archived
Owner:
Closed: Aug 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

x86-generic-tot-asan-informational failures in tpm_manager (odr-violation) and attestation (leaks)

Project Member Reported by jamescook@chromium.org, Aug 18 2016

Issue description

This is breaking all builds since ~9 PM last night.

Last good x86 build:
https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-tot-asan-informational/builds/11699

First bad x86 build:
https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-tot-asan-informational/builds/11700

Example bad amd64 build (has better logs, there were other infra issues that mean I can't tell the first bad build here):
https://build.chromium.org/p/chromiumos.chromium/builders/amd64-generic-tot-asan-informational/builds/10274

Strangely, the code lives in src/aosp/system/tpm and that repo hasn't changed since August 9.

ODR violation message:
tpm_manager-0.0.1-r471:  * ERROR: chromeos-base/tpm_manager-0.0.1-r471::chromiumos failed (test phase):
<snip>
tpm_manager-0.0.1-r471:  * ASAN error detected:
tpm_manager-0.0.1-r471:  * =================================================================
tpm_manager-0.0.1-r471:  * ==17==ERROR: AddressSanitizer: odr-violation (0x7ff47370e8c0):
tpm_manager-0.0.1-r471:  *   [1] size=8 'tpm_manager::NvramPolicyRecord::default_instance_' gen/include/tpm_manager/common/tpm_manager.pb.cc:237:39
tpm_manager-0.0.1-r471:  *   [2] size=8 'tpm_manager::NvramPolicyRecord::default_instance_' gen/include/tpm_manager/common/tpm_manager.pb.cc:237:39
tpm_manager-0.0.1-r471:  * These globals were registered at these points:
tpm_manager-0.0.1-r471:  *   [1]:
tpm_manager-0.0.1-r471:  *     #0 0x7ff47252c728 in __asan_register_globals.part.14 (/var/cache/portage/chromeos-base/tpm_manager/out/Default/tpm_manager_testrunner+0x1cc728)
tpm_manager-0.0.1-r471:  *     #1 0x7ff47273ae1b in asan.module_ctor (/var/cache/portage/chromeos-base/tpm_manager/out/Default/tpm_manager_testrunner+0x3dae1b)
tpm_manager-0.0.1-r471:  *     #2 0x7ff4705a4f44 in __libc_start_main /var/tmp/portage/cross-x86_64-cros-linux-gnu/glibc-2.19-r12/work/glibc-2.19/csu/libc-start.c:251
tpm_manager-0.0.1-r471:  *     #3 0x7ff47251baf3 in _start (/var/cache/portage/chromeos-base/tpm_manager/out/Default/tpm_manager_testrunner+0x1bbaf3)
tpm_manager-0.0.1-r471:  * 
tpm_manager-0.0.1-r471:  *   [2]:
tpm_manager-0.0.1-r471:  *     #0 0x7ff47252c728 in __asan_register_globals.part.14 (/var/cache/portage/chromeos-base/tpm_manager/out/Default/tpm_manager_testrunner+0x1cc728)
tpm_manager-0.0.1-r471:  *     #1 0x7ff4722dd12b in asan.module_ctor (/var/cache/portage/chromeos-base/tpm_manager/out/Default/lib/libtpm_manager.so+0xdc12b)
tpm_manager-0.0.1-r471:  *     #2 0x7ff47214b8ab in call_init /var/tmp/portage/cross-x86_64-cros-linux-gnu/glibc-2.19-r12/work/glibc-2.19/elf/dl-init.c:36
tpm_manager-0.0.1-r471:  *     #3 0x7ff47214b8ab in _dl_init /var/tmp/portage/cross-x86_64-cros-linux-gnu/glibc-2.19-r12/work/glibc-2.19/elf/dl-init.c:126
tpm_manager-0.0.1-r471:  *     #4 0x7ff47213cd99  (/lib64/ld-linux-x86-64.so.2+0xd99)

Example leak:
attestation-0.0.1-r1188:  * ==17==ERROR: LeakSanitizer: detected memory leaks
attestation-0.0.1-r1188:  * 
attestation-0.0.1-r1188:  * Direct leak of 72 byte(s) in 1 object(s) allocated from:
attestation-0.0.1-r1188:  *     #0 0x7fe03107b740 in operator new(unsigned long) (/var/cache/portage/chromeos-base/attestation/out/Default/attestation_testrunner+0x5a6740)
attestation-0.0.1-r1188:  *     #1 0x7fe03063137e in base::internal::PostTaskAndReplyImpl::PostTaskAndReply(tracked_objects::Location const&, base::Callback<void (), (base::internal::CopyMode)1> const&, base::Callback<void (), (base::internal::CopyMode)1> const&) /build/amd64-generic/tmp/portage/chromeos-base/libchrome-395517-r3/work/libchrome-395517/base/threading/post_task_and_reply_impl.cc:82
attestation-0.0.1-r1188:  *     #2 0x7fe03062a743 in base::TaskRunner::PostTaskAndReply(tracked_objects::Location const&, base::Callback<void (), (base::internal::CopyMode)1> const&, base::Callback<void (), (base::internal::CopyMode)1> const&) /build/amd64-generic/tmp/portage/chromeos-base/libchrome-395517-r3/work/libchrome-395517/base/task_runner.cc:53
attestation-0.0.1-r1188:  *     #3 0x7fe03152cfe8 in attestation::AttestationService::CreateGoogleAttestedKey(attestation::CreateGoogleAttestedKeyRequest const&, base::Callback<void (attestation::CreateGoogleAttestedKeyReply const&), (base::internal::CopyMode)1> const&) /build/amd64-generic/var/cache/portage/chromeos-base/attestation/out/Default/../../../../../../../tmp/portage/chromeos-base/attestation-0.0.1-r1188/work/attestation-0.0.1/aosp/system/tpm/attestation/server/attestation_service.cc:90:3
attestation-0.0.1-r1188:  *     #4 0x7fe031153206 in attestation::AttestationServiceTest_CreateGoogleAttestedKeyAndCancel_Test::TestBody() /build/amd64-generic/var/cache/portage/chromeos-base/attestation/out/Default/../../../../../../../tmp/portage/chromeos-base/attestation-0.0.1-r1188/work/attestation-0.0.1/aosp/system/tpm/attestation/server/attestation_service_test.cc:418:3
attestation-0.0.1-r1188:  *     #5 0x7fe02fafb7b7 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /build/amd64-generic/tmp/portage/dev-cpp/gtest-1.7.0/work/gtest-1.7.0-.amd64/./src/gtest.cc:2078
attestation-0.0.1-r1188:  *     #6 0x7fe02fafb7b7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /build/amd64-generic/tmp/portage/dev-cpp/gtest-1.7.0/work/gtest-1.7.0-.amd64/./src/gtest.cc:2114
attestation-0.0.1-r1188:  *     #7 0x7fe02faf041e in testing::Test::Run() /build/amd64-generic/tmp/portage/dev-cpp/gtest-1.7.0/work/gtest-1.7.0-.amd64/./src/gtest.cc:2151
attestation-0.0.1-r1188:  *     #8 0x7fe02faf04d3 in testing::TestInfo::Run() /build/amd64-generic/tmp/portage/dev-cpp/gtest-1.7.0/work/gtest-1.7.0-.amd64/./src/gtest.cc:2326
attestation-0.0.1-r1188:  *     #9 0x7fe02faf05e4 in testing::TestCase::Run() /build/amd64-generic/tmp/portage/dev-cpp/gtest-1.7.0/work/gtest-1.7.0-.amd64/./src/gtest.cc:2444
attestation-0.0.1-r1188:  *     #10 0x7fe02faf08c7 in testing::internal::UnitTestImpl::RunAllTests() /build/amd64-generic/tmp/portage/dev-cpp/gtest-1.7.0/work/gtest-1.7.0-.amd64/./src/gtest.cc:4315
attestation-0.0.1-r1188:  *     #11 0x7fe02faf0bc1 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /build/amd64-generic/tmp/portage/dev-cpp/gtest-1.7.0/work/gtest-1.7.0-.amd64/./src/gtest.cc:2078
attestation-0.0.1-r1188:  *     #12 0x7fe02faf0bc1 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /build/amd64-generic/tmp/portage/dev-cpp/gtest-1.7.0/work/gtest-1.7.0-.amd64/./src/gtest.cc:2114
attestation-0.0.1-r1188:  *     #13 0x7fe02faf0bc1 in testing::UnitTest::Run() /build/amd64-generic/tmp/portage/dev-cpp/gtest-1.7.0/work/gtest-1.7.0-.amd64/./src/gtest.cc:3929
attestation-0.0.1-r1188:  *     #14 0x7fe03107e57d in RUN_ALL_TESTS() /build/amd64-generic/usr/include/gtest/gtest.h:2288:10
attestation-0.0.1-r1188:  *     #15 0x7fe03107e57d in main /build/amd64-generic/var/cache/portage/chromeos-base/attestation/out/Default/../../../../../../../tmp/portage/chromeos-base/attestation-0.0.1-r1188/work/attestation-0.0.1/aosp/system/tpm/attestation/attestation_testrunner.cc:30
attestation-0.0.1-r1188:  *     #16 0x7fe02eac3fb5 in __libc_start_main /var/tmp/portage/cross-x86_64-cros-linux-gnu/glibc-2.19-r12/work/glibc-2.19/csu/libc-start.c:292
attestation-0.0.1-r1188:  *     #17 0x7fe030fa0793 in _start (/var/cache/portage/chromeos-base/attestation/out/Default/attestation_testrunner+0x4cb793)

 
Cc: apronin@chromium.org
+apronin

This failure correlates in time with several attestation related changes that landed last night:

"cryptohome: add attestation dependency"
https://chromium-review.googlesource.com/#/c/371600/

"cryptohome: Extract attestation into dbus service child class" https://chromium-review.googlesource.com/#/c/368890/

"cryptohome: add empty skeleton for ServiceDistributed"
https://chromium-review.googlesource.com/#/c/368891/

Could any of these be related?  I would speculatively revert, but I'm not sure which.

Cc: -apronin@chromium.org sureshraj@chromium.org benzh@chromium.org
Owner: apronin@chromium.org
Status: Assigned (was: Untriaged)
It looks like tpm_manager wasn't being built/run as part of ASAN unit tests on the green builds, but it starts being built/run on the red builds.

This makes me very suspicious of "cryptohome: add attestation dependency"
https://chromium-review.googlesource.com/#/c/371600/

Is that safe to revert?

Yes, it must be because of that CL: https://chromium-review.googlesource.com/#/c/371600/.
There are several ways to address, and the real fix is fixing the tpm_manager leak. But, if needed, for now it's safe to revert it - we won't need it until a later pack of CLs which is not merged or submitted to CQ yet.

Comment 4 by benzh@chromium.org, Aug 18 2016

@apronin could you chump the revert of the CL since it's breaking the ASAN builder? We can land the real fix when it's ready.
Reverted with https://chromium-review.googlesource.com/#/c/372266/

Just FYI, apparently chrome and libchromeos-ui suppress the odr-violation warning, see  issue 607223 . However, the leaks are still worrisome.

Searching cs/ for detect_odr_violation also turns up a suppression in android.

Thanks for the help.

Status: Fixed (was: Assigned)

Comment 7 by dchan@chromium.org, Oct 7 2016

Labels: VerifyIn-55

Comment 8 by dchan@google.com, Nov 19 2016

Labels: VerifyIn-56

Comment 9 by dchan@google.com, Jan 21 2017

Labels: VerifyIn-57

Comment 10 by dchan@google.com, Mar 4 2017

Labels: VerifyIn-58

Comment 11 by dchan@google.com, Apr 17 2017

Labels: VerifyIn-59

Comment 12 by dchan@google.com, May 30 2017

Labels: VerifyIn-60
Labels: VerifyIn-61

Comment 14 by dchan@chromium.org, Oct 14 2017

Status: Archived (was: Fixed)

Sign in to add a comment