x86-generic-tot-asan-informational failing vmtests |
||||
Issue descriptionLink to builds https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-tot-asan-informational/ Starting from build #10344 we're getting past build but failing in a subset of VM tests. Pri-2 because this is failing an informational bot, not blocking PFQ
,
Apr 27 2016
The test was last passing at build #10291. First failing at https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-tot-asan-informational/builds/10292
,
Apr 27 2016
On chromeos side there was only 2 CLs between the passing build #10291 and failing build #10292: https://chromium-review.googlesource.com/#/c/340470/ chromeos-chrome: switch 32 bit to component build. https://chromium-review.googlesource.com/#/c/338212/ Add Samsung K4E6E304EB 4G LPDDR3 SDRAM for elm-rev1 SKU2
,
Apr 27 2016
There are 8 chromium CLs listed under https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-tot-asan-informational/builds/10292 but none look suspicious. Another important observation is that we don't have these failures on amd64-generic-tot-asan-informational (https://build.chromium.org/p/chromiumos.chromium/builders/amd64-generic-tot-asan-informational)
,
Apr 27 2016
So I have been trying to reproduce this locally. If I just build x86-generic with asan (setup_board with profile=asan), and use that to USB-boot a device (e.g. alex), it will just crash before Chrome login screen is shown. So I guess this counts as a repro, although it is difficult to debug. I also tried to play with the GYP_DEFINES (set asan=1 and component=shared_library) under simplechrome but somehow the build always fails with errors like x86_64-cros-linux-gnu-gcc.real.elf: error: find-bad-constructs: No such file or directory x86_64-cros-linux-gnu-gcc.real.elf: error: check-templates: No such file or directory x86_64-cros-linux-gnu-gcc.real.elf: error: follow-macro-expansion: No such file or directory x86_64-cros-linux-gnu-gcc.real.elf: error: check-implicit-copy-ctors: No such file or directory x86_64-cros-linux-gnu-gcc.real.elf: error: unrecognized command line option '-Xclang' x86_64-cros-linux-gnu-gcc.real.elf: error: unrecognized command line option '-Xclang' x86_64-cros-linux-gnu-gcc.real.elf: error: unrecognized command line option '-Xclang' x86_64-cros-linux-gnu-gcc.real.elf: error: unrecognized command line option '-add-plugin' x86_64-cros-linux-gnu-gcc.real.elf: error: unrecognized command line option '-Xclang' x86_64-cros-linux-gnu-gcc.real.elf: error: unrecognized command line option '-Xclang' x86_64-cros-linux-gnu-gcc.real.elf: error: unrecognized command line option '-plugin-arg-find-bad-constructs' x86_64-cros-linux-gnu-gcc.real.elf: error: unrecognized command line option '-Xclang' x86_64-cros-linux-gnu-gcc.real.elf: error: unrecognized command line option '-Xclang' Will try a 32-bit build under simplechrome next
,
Apr 27 2016
=================================================================
==3686==ERROR: AddressSanitizer: odr-violation (0x6ce055c0):
[1] size=24 'vtable for ui::ClientNativePixmapFactory' ../../../../../../../home/chrome-bot/chrome_root/src/ui/ozone/public/client_native_pixmap_factory.cc
[2] size=24 'vtable for ui::ClientNativePixmapFactory' ../../../../../../../home/chrome-bot/chrome_root/src/ui/ozone/public/client_native_pixmap_factory.cc
These globals were registered at these points:
[1]:
#0 0x55d0db82 (/opt/google/chrome/chrome+0x724b82)
#1 0x6cc55a4f (/opt/google/chrome/lib/libgpu.so+0x534a4f)
#2 0x77774ac1 (/lib/ld-linux.so.2+0x10ac1)
#3 0x77764b1e (/lib/ld-linux.so.2+0xb1e)
[2]:
#0 0x55d0db82 (/opt/google/chrome/chrome+0x724b82)
#1 0x6332492f (/opt/google/chrome/lib/libozone.so+0x1092f)
#2 0x77774ac1 (/lib/ld-linux.so.2+0x10ac1)
#3 0x77764b1e (/lib/ld-linux.so.2+0xb1e)
==3686==HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_odr_violation=0
SUMMARY: AddressSanitizer: odr-violation: global 'vtable for ui::ClientNativePixmapFactory' at ../../../../../../../home/chrome-bot/chrome_root/src/ui/ozone/public/client_native_pixmap_factory.cc
==3686==ABORTING
,
Apr 27 2016
re:#6 thanks I finally managed to get into frecon without the system rebooting, and my /var/log/chrome/asan_log has tons of the same warnings. That at least confirms I have a local repro. I still can't build with simplechrome, however, but I would be able to build from inside chroot.
,
Apr 27 2016
Patch uploaded to https://chromium-review.googlesource.com/#/c/341080/1 Thanks to ihf@ for helping debug this!
,
Apr 28 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform2/+/b355d37a63088dfdc17dc07b6bd4ca2ef60125a1 commit b355d37a63088dfdc17dc07b6bd4ca2ef60125a1 Author: Haixia Shi <hshi@chromium.org> Date: Wed Apr 27 23:07:11 2016 libchromeos-ui: add ASAN_OPTIONS=detect_odr_violation=0 With component build enabled, we are getting a lot of odr-violation errors for vtables between different component libraries. By default chrome disables this option, so we shall do the same on Chrome OS. BUG= chromium:607223 TEST=emerge libchromeos-ui, verify no more asan errors Change-Id: I58ec81c28ff81bc007a619d4ff489cf0d540518e Reviewed-on: https://chromium-review.googlesource.com/341080 Trybot-Ready: Haixia Shi <hshi@chromium.org> Tested-by: Haixia Shi <hshi@chromium.org> Reviewed-by: Ilja Friedel <ihf@chromium.org> Commit-Queue: Haixia Shi <hshi@chromium.org> [modify] https://crrev.com/b355d37a63088dfdc17dc07b6bd4ca2ef60125a1/libchromeos-ui/chromeos/ui/chromium_command_builder.cc
,
Apr 28 2016
,
Apr 28 2016
Now Passing https://build.chromium.org/p/chromiumos.chromium/builders/x86-generic-tot-asan-informational/builds/10354 |
||||
►
Sign in to add a comment |
||||
Comment 1 by h...@chromium.org
, Apr 27 2016Sample log /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-login_OwnershipRetaken [ FAILED ] /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-login_OwnershipRetaken ERROR: Timed out waiting for login prompt /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-login_OwnershipRetaken/login_OwnershipRetaken [ FAILED ] /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-login_OwnershipRetaken/login_OwnershipRetaken ERROR: Timed out waiting for login prompt /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-login_OwnershipRetaken/login_OwnershipRetaken 04/27 09:08:43.293 ERROR| site_utils:0236| Timed out waiting for login prompt /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-login_OwnershipRetaken/login_OwnershipRetaken 04/27 09:08:43.328 ERROR| test:0613| Ignoring exception during cleanup() phase: /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-login_OwnershipRetaken/login_OwnershipRetaken 04/27 09:08:43.332 ERROR| traceback:0013| Traceback (most recent call last): /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-login_OwnershipRetaken/login_OwnershipRetaken 04/27 09:08:43.339 ERROR| traceback:0013| File "/usr/local/autotest/common_lib/test.py", line 611, in _exec /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-login_OwnershipRetaken/login_OwnershipRetaken 04/27 09:08:43.343 ERROR| traceback:0013| _cherry_pick_call(self.cleanup, *args, **dargs) /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-login_OwnershipRetaken/login_OwnershipRetaken 04/27 09:08:43.346 ERROR| traceback:0013| File "/usr/local/autotest/common_lib/test.py", line 719, in _cherry_pick_call /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-login_OwnershipRetaken/login_OwnershipRetaken 04/27 09:08:43.350 ERROR| traceback:0013| return func(*p_args, **p_dargs) /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-login_OwnershipRetaken/login_OwnershipRetaken 04/27 09:08:43.353 ERROR| traceback:0013| File "/usr/local/autotest/tests/login_OwnershipRetaken/login_OwnershipRetaken.py", line 91, in cleanup /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-login_OwnershipRetaken/login_OwnershipRetaken 04/27 09:08:43.356 ERROR| traceback:0013| cros_ui.restart() /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-login_OwnershipRetaken/login_OwnershipRetaken 04/27 09:08:43.360 ERROR| traceback:0013| File "/usr/local/autotest/cros/cros_ui.py", line 130, in restart /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-login_OwnershipRetaken/login_OwnershipRetaken 04/27 09:08:43.363 ERROR| traceback:0013| raise error.TestError('Could not stop session') /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-login_OwnershipRetaken/login_OwnershipRetaken 04/27 09:08:43.367 ERROR| traceback:0013| TestError: Could not stop session /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-01-login_OwnershipRetaken/login_OwnershipRetaken 04/27 09:08:43.370 ERROR| test:0617| Now raising the earlier <class 'autotest_lib.client.bin.site_utils.TimeoutError'> error /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-02-login_MultiUserPolicy [ FAILED ] /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-02-login_MultiUserPolicy ERROR: Timed out waiting for login prompt /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-02-login_MultiUserPolicy/login_MultiUserPolicy [ FAILED ] /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-02-login_MultiUserPolicy/login_MultiUserPolicy ERROR: Timed out waiting for login prompt /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-02-login_MultiUserPolicy/login_MultiUserPolicy 04/27 09:10:24.625 ERROR| site_utils:0236| Timed out waiting for login prompt /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-03-platform_CrosDisksDBus [ FAILED ] /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-03-platform_CrosDisksDBus FAIL: Unhandled DBusException: org.freedesktop.DBus.Error.ServiceUnknown: The name org.chromium.CrosDisks was not provided by any .service files /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-03-platform_CrosDisksDBus/platform_CrosDisksDBus [ FAILED ] /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-03-platform_CrosDisksDBus/platform_CrosDisksDBus FAIL: Unhandled DBusException: org.freedesktop.DBus.Error.ServiceUnknown: The name org.chromium.CrosDisks was not provided by any .service files /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-04-security_EnableChromeTesting [ FAILED ] /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-04-security_EnableChromeTesting ERROR: Test subprocess failed rc=256 /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-04-security_EnableChromeTesting/security_EnableChromeTesting [ FAILED ] /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-04-security_EnableChromeTesting/security_EnableChromeTesting ERROR: Test subprocess failed rc=256 /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-04-security_EnableChromeTesting/security_EnableChromeTesting 04/27 09:10:50.420 ERROR| log:0072| post-test siteration sysinfo error: /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-04-security_EnableChromeTesting/security_EnableChromeTesting 04/27 09:10:50.424 ERROR| traceback:0013| Traceback (most recent call last): /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-04-security_EnableChromeTesting/security_EnableChromeTesting 04/27 09:10:50.428 ERROR| traceback:0013| File "/usr/local/autotest/common_lib/log.py", line 70, in decorated_func .... .... /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-50-login_RetrieveActiveSessions [ FAILED ] /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-50-login_RetrieveActiveSessions ERROR: Could not stop session /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-50-login_RetrieveActiveSessions/login_RetrieveActiveSessions [ FAILED ] /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-50-login_RetrieveActiveSessions/login_RetrieveActiveSessions ERROR: Could not stop session /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-51-login_SameSessionTwice [ FAILED ] /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-51-login_SameSessionTwice ERROR: Timed out waiting for login prompt /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-51-login_SameSessionTwice/login_SameSessionTwice [ FAILED ] /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-51-login_SameSessionTwice/login_SameSessionTwice ERROR: Timed out waiting for login prompt /tmp/cbuildbotheKoxU/test_harness/all/SimpleTestVerify/1_autotest_tests/results-51-login_SameSessionTwice/login_SameSessionTwice 04/27 09:41:15.008 ERROR| site_utils:0236| Timed out waiting for login prompt -------------------------------------------------------------------------------------------------------------------------------------------------------------- Total PASS: 58/105 (55%)