Linux Chromium OS ASan LSan Tests (1) consistently fails MANY tests |
||||
Issue descriptionBot: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Chromium%20OS%20ASan%20LSan%20Tests%20%281%29 First failure: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Chromium%20OS%20ASan%20LSan%20Tests%20%281%29/28394 Failure app_list_unittests Failure app_shell_unittests Failure ash_content_unittests Failure ash_unittests Failure aura_unittests Failure browser_tests Failure viz_browser_tests Failure cc_unittests Failure chromevox_tests Failure components_browsertests Failure components_unittests Failure compositor_unittests Failure content_browsertests Failure viz_content_browsertests Failure content_unittests Failure viz_content_unittests Failure exo_unittests Failure extensions_browsertests Failure gl_unittests_ozone Failure gpu_unittests Failure interactive_ui_tests Failure keyboard_unittests Failure media_unittests Failure message_center_unittests Failure mojo_core_unittests Failure ozone_gl_unittests Failure select_to_speak_extension_tests Failure services_unittests Failure snapshot_unittests Failure sync_integration_tests Failure ui_chromeos_unittests Failure unit_tests Failure views_mus_interactive_ui_tests Failure views_mus_unittests Failure views_unittests Failure viz_unittests Failure wayland_client_perftests Failure wm_unittests It is unclear to me from the blamelist which CL is the culprit, and I cannot reproduce the failures with a local ChromeOS ASan/LSan build, so I can't bisect. "Findit is Analyzing...", no results yet: https://findit-for-me.appspot.com/waterfall/failure?url=https://build.chromium.org/p/chromium.memory/builders/Linux%20Chromium%20OS%20ASan%20LSan%20Tests%20%281%29/builds/28396
,
Jul 26
Copying a few notes from issue 867786 : kylechar@: """ Wait, nevermind, the bot has caught up and the revert was included in this run: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Chromium%20OS%20ASan%20LSan%20Tests%20%281%29/28398 That didn't fix the problem. https://crrev.com/c/1150662 wasn't the issue, so I'll reland it. Here is the first failing run, it must be something else from that: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Chromium%20OS%20ASan%20LSan%20Tests%20%281%29/28394 """ kylechar@: """ I don't see any other CLs in the suspect change list that are obviously the culprit. The failure is CrOS only so probably a change to CrOS code. There are a couple CLs that touch CrOS code tho. I tried running some of the failing tests with ASan at ToT (and ToT plus https://crrev.com/c/1150662) and they're not failing, so maybe the problem was fixed? I'm not going to look at this any further, it's not https://crrev.com/c/1150662 and I can't reproduce. """
,
Jul 26
I'm going to poke at this.
,
Jul 26
Interesting enough I can't reproduce this locally on tip of tree.
,
Jul 26
I can't even reproduce when I go back to the first failure. I'm wondering if something is weird with the bots running the tests.
,
Jul 26
Still poking at this. Can't repro at all locally, or when using 'mb' as the bots do. Latest thinking is perhaps it's a mismatch between build libraries and libraries on the bots?
,
Jul 26
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/c2b25737b2f878e0917d8cd0553d11c557b5cb92 commit c2b25737b2f878e0917d8cd0553d11c557b5cb92 Author: Scott Violet <sky@chromium.org> Date: Thu Jul 26 21:00:10 2018 Revert "Remove libcpp_is_static" This reverts commit f002a96e9b788fe71fd1c773a4bc891940c409d8. Reason for revert: Dirk bisected the chromeos asan failures and it appears this is the culprit. The output from the failing bots isn't particularly helpful as it doesn't have symbols. Here's a recent failure: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Chromium%20OS%20ASan%20LSan%20Tests%20%281%29/28402 BUG= 867826 Original change's description: > Remove libcpp_is_static > > This change makes libc++ a shared_library iff is_component_build=true, just like > all other components. This has the advantage that the RPATH doesn't need to be > set in sanitizer (non-component) builds just so binaries can find libc++.so. > This CL also changes RPATH to be set iff is_component_build=true. > > Needs buildtools CL: > https://chromium-review.googlesource.com/c/chromium/buildtools/+/1142370 > > TBR=dpranke,jochen > R=thakis > BUG= 850682 > CQ_INCLUDE_TRYBOTS=luci.chromium.try:linux_chromium_msan_rel_ng;master.tryserver.chromium.linux:linux_chromium_ubsan_rel_ng;luci.chromium.try:linux_chromium_asan_rel_ng;luci.chromium.try:linux_chromium_tsan_rel_ng > > Change-Id: I96e74da6d907659b939a0f229693bbd2a6d9140a > Reviewed-on: https://chromium-review.googlesource.com/1119098 > Commit-Queue: Thomas Anderson <thomasanderson@chromium.org> > Reviewed-by: Peter Collingbourne <pcc@chromium.org> > Reviewed-by: Nico Weber <thakis@chromium.org> > Cr-Commit-Position: refs/heads/master@{#578085} TBR=thakis@chromium.org,pcc@chromium.org,thomasanderson@chromium.org Change-Id: Ieec9ae83b7c12004521358cf71eb2d09fdfc7a09 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 850682 Cq-Include-Trybots: luci.chromium.try:linux_chromium_msan_rel_ng;master.tryserver.chromium.linux:linux_chromium_ubsan_rel_ng;luci.chromium.try:linux_chromium_asan_rel_ng;luci.chromium.try:linux_chromium_tsan_rel_ng Reviewed-on: https://chromium-review.googlesource.com/1151917 Reviewed-by: Scott Violet <sky@chromium.org> Commit-Queue: Scott Violet <sky@chromium.org> Cr-Commit-Position: refs/heads/master@{#578435} [modify] https://crrev.com/c2b25737b2f878e0917d8cd0553d11c557b5cb92/DEPS [modify] https://crrev.com/c2b25737b2f878e0917d8cd0553d11c557b5cb92/build/config/BUILD.gn [modify] https://crrev.com/c2b25737b2f878e0917d8cd0553d11c557b5cb92/build/config/BUILDCONFIG.gn [modify] https://crrev.com/c2b25737b2f878e0917d8cd0553d11c557b5cb92/build/config/c++/c++.gni [modify] https://crrev.com/c2b25737b2f878e0917d8cd0553d11c557b5cb92/build/config/gcc/BUILD.gn [modify] https://crrev.com/c2b25737b2f878e0917d8cd0553d11c557b5cb92/build/config/posix/BUILD.gn [modify] https://crrev.com/c2b25737b2f878e0917d8cd0553d11c557b5cb92/build/config/win/BUILD.gn [modify] https://crrev.com/c2b25737b2f878e0917d8cd0553d11c557b5cb92/chrome/installer/linux/BUILD.gn [modify] https://crrev.com/c2b25737b2f878e0917d8cd0553d11c557b5cb92/extensions/shell/installer/linux/BUILD.gn [modify] https://crrev.com/c2b25737b2f878e0917d8cd0553d11c557b5cb92/remoting/host/linux/BUILD.gn
,
Jul 26
Dirk bisected and apparently it was Tom's patch. Revert is cycling through now. Will keep open until we know for sure if the revert fixed it.
,
Jul 27
The bot is still red, but the current issues look unrelated to this. Marking as fixed, and following up on remaining failures. |
||||
►
Sign in to add a comment |
||||
Comment 1 by smcgruer@chromium.org
, Jul 26