The build is setting symbol_level=1 in the GN args, so it's not clear why the backtraces in the failing test do not include symbol names.

I noticed after filling this that it's not an ASAN failure stack but rather a "process termination" stack. Not sure if that changes anything (same stack did symbolize on Linux waterfall (non ASAN) when it flaked similarly there : https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Tests%20%28dbg%29%281%29%2832%29/52302 ).

This is apparently working as intended. Back in March https://crrev.com/c/940771 was submitted that changes the stack trace to use unsymbolized stack traces when using asan, lsan, msan or tsan, with the expectation that you can run tools/valgrind/asan/asan_symbolize.py to get a better stack trace that includes more details (file name, line numbers).

+jdarpinian@ from that change. It's highly impractical to not have symbols on failures on the try bots.. (especially as a Windows dev). The unsymbolized log should at the very least refer to this build flag for someone that cares to force the bot to print more information (in this case the failure did not reproduce locally..). 

Can the bot automatically run tools/valgrind/asan/asan_symbolize.py if this is required to get a symbolized callstack?

 Issue 887543  has been merged into this issue.

Comment 4 is incorrect, that change is behavior-preserving. asan builds already weren't symbolized in-process because they run (or are supposed to run) through an external symbolizer. That change just added a gn arg so people could opt in to that in non-asan builds.

The bots should transparently run said external symbolizer though, and they should print symoblized output -- e.g. comment 7 is how things are supposed to work.

Does anyone have an idea when this regressed?

Looking at https://chromium-swarm.appspot.com/task?id=3fe8cd87f0846010&refresh=10&show_raw=1 :

It seems to be running testing/test_env.py as it should (output is from https://cs.chromium.org/chromium/src/testing/test_env.py?q=test_env.py&sq=package:chromium&dr&l=268).

That's supposed to symbolize: https://cs.chromium.org/chromium/src/testing/test_env.py?q=test_env.py&sq=package:chromium&dr&l=292

Looking at the builder: https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Chromium%20OS%20ASan%20LSan%20Builder/41016

dcheck_always_on = true
is_asan = true
is_component_build = false
is_debug = false
is_lsan = true
strip_absolute_paths_from_debug_symbols = true
symbol_level = 1
target_os = "chromeos"
use_goma = true
goma_dir = "/b/swarming/w/ir/cache/goma/client"


symbol_level=1 should be enough to get stacks.


https://chromium-review.googlesource.com/c/chromium/src/+/1164532 landed around the timeframe when this got filed and is in the area, so maybe it's somehow related (but it doesn't really look like that to me).


Next step is probably to download the isolate from https://chromium-swarm.appspot.com/task?id=3fe8cd87f0846010&refresh=10&show_raw=1 and run test_env.py locally like the isolated file would and see what goes wrong.

As thakis@ noted, my commit did not change the behavior of ASAN builds.

Okay, so after downloading the the isolate from https://chromium-swarm.appspot.com/task?id=404a3368c4692f10&refresh=10&request_detail=true&show_raw=1 for build https://ci.chromium.org/p/chromium/builders/luci.chromium.try/linux_chromium_asan_rel_ng/110615 (it's a much quicker example to run), I reproed the failure by running "GTEST_SHARD_INDEX=1 GTEST_TOTAL_SHARDS=5 ../../testing/xvfb.py ./components_unittests --brave-new-test-launcher --test-launcher-bot-mode --asan=1 --msan=0 --tsan=0 --cfi-diag=0 --lsan=1" from the out/Release directory inside of the isolate download. The backtraces from the sanitizer are downloaded, but the backtraces resulting from the EXPECT_CALL failures are not.

Before doing that I had run it without the --lsan=1 flag because I was not aware of the "Extra Args" for the swarming task. Doing that, even the EXPECT_CALL failure backtraces were symbolized. Looking at https://cs.chromium.org/chromium/src/testing/test_env.py?rcl=a008c590370f9e0549907ba5d79ae157598432be&l=253, it looks like when LSan is enabled when running the test, it doesn't use the external symbolization script and instead relies on the in-process symbolizer. Does the in-process symbolizer get run for all backtraces or just for those created by the sanitizer?

In comment 13, where I said the backtraces are downloaded, I meant symbolized not downloaded.

Shouldn't this bit here cause lsan builds to call the symbolization script (even though in-process):

72	    # LSan is not sandbox-compatible, so we can use online symbolization. In
73	    # fact, it needs symbolization to be able to apply suppressions.
74	    symbolization_options = ['symbolize=1',
75	                             'external_symbolizer_path=%s' % symbolizer_path]

?

I don't know, I have no experience with the sanitizers, but given that those values are part of the ASAN_OPTIONS environment variable, I could totally envision them not doing anything to stack traces that aren't created by one of the sanitizers (sanitizer backtraces are symbolized when LSan is enabled).

I spent some time stracing this, and it appears my guess is correct. The symbolizer script is only called by the sanitizer, it's not executed in the case of an EXPECT_CALL failure. I don't know what (if anything) should be done to fix this or who would be best to fix it.

This bug looks like something is wrong in chromium/src side scripts. I think this is a sheriff bug, not a trooper bug.

Not sure what a good component is for this, unfortunately.

While I'd agree that this isn't a trooper bug, I>C>C might still be the best place for it. I don't think this is a sheriff bug.