browser_tests and content_browsertests red on ClangToTMacASan_tester |
||||||||||
Issue descriptionStarted in https://build.chromium.org/p/chromium.fyi/builders/ClangToTMacASan%20tester/builds/3746 https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.fyi%2FClangToTMacASan_tester%2F3746%2F%2B%2Frecipes%2Fsteps%2Fbrowser_tests%2F0%2Fstdout https://luci-logdog.appspot.com/v/?s=chromium%2Fbb%2Fchromium.fyi%2FClangToTMacASan_tester%2F3746%2F%2B%2Frecipes%2Fsteps%2Fcontent_browsertests%2F0%2Fstdout Looks like this is the cause: Traceback (most recent call last): File "/b/swarm_slave/w/irjUM_0i/tools/valgrind/asan/asan_symbolize.py", line 271, in <module> main() File "/b/swarm_slave/w/irjUM_0i/tools/valgrind/asan/asan_symbolize.py", line 268, in main loop.process_logfile() File "/b/swarm_slave/w/irjUM_0i/tools/valgrind/asan/third_party/asan_symbolize.py", line 416, in process_logfile processed = self.process_line(line) File "/b/swarm_slave/w/irjUM_0i/tools/valgrind/asan/third_party/asan_symbolize.py", line 439, in process_line_posix symbolized_line = self.symbolize_address(addr, binary, offset) File "/b/swarm_slave/w/irjUM_0i/tools/valgrind/asan/third_party/asan_symbolize.py", line 393, in symbolize_address result = symbolizers[binary].symbolize(addr, binary, offset) File "/b/swarm_slave/w/irjUM_0i/tools/valgrind/asan/third_party/asan_symbolize.py", line 244, in symbolize result = symbolizer.symbolize(addr, binary, offset) File "/b/swarm_slave/w/irjUM_0i/tools/valgrind/asan/third_party/asan_symbolize.py", line 216, in symbolize atos_line = self.atos.convert('0x%x' % int(offset, 16)) File "/b/swarm_slave/w/irjUM_0i/tools/valgrind/asan/third_party/asan_symbolize.py", line 192, in convert self.w.write(line + "\n") IOError: [Errno 5] Input/output error Error: failed to symbolize snippets in JSON: Traceback (most recent call last): File "/b/swarm_slave/w/irjUM_0i/tools/valgrind/asan/asan_symbolize.py", line 271, in <module> main() File "/b/swarm_slave/w/irjUM_0i/tools/valgrind/asan/asan_symbolize.py", line 264, in main symbolize_snippets_in_json(args.test_summary_json_file, loop) File "/b/swarm_slave/w/irjUM_0i/tools/valgrind/asan/asan_symbolize.py", line 227, in symbolize_snippets_in_json test_run_symbolizer.symbolize(test_run) File "/b/swarm_slave/w/irjUM_0i/tools/valgrind/asan/asan_symbolize.py", line 199, in symbolize symbolized_snippet = self.symbolize_snippet(original_snippet) File "/b/swarm_slave/w/irjUM_0i/tools/valgrind/asan/asan_symbolize.py", line 194, in symbolize_snippet symbolized_lines += self.symbolization_loop.process_line(line) File "/b/swarm_slave/w/irjUM_0i/tools/valgrind/asan/third_party/asan_symbolize.py", line 439, in process_line_posix symbolized_line = self.symbolize_address(addr, binary, offset) File "/b/swarm_slave/w/irjUM_0i/tools/valgrind/asan/third_party/asan_symbolize.py", line 393, in symbolize_address result = symbolizers[binary].symbolize(addr, binary, offset) File "/b/swarm_slave/w/irjUM_0i/tools/valgrind/asan/third_party/asan_symbolize.py", line 244, in symbolize result = symbolizer.symbolize(addr, binary, offset) File "/b/swarm_slave/w/irjUM_0i/tools/valgrind/asan/third_party/asan_symbolize.py", line 216, in symbolize atos_line = self.atos.convert('0x%x' % int(offset, 16)) File "/b/swarm_slave/w/irjUM_0i/tools/valgrind/asan/third_party/asan_symbolize.py", line 192, in convert self.w.write(line + "\n") IOError: [Errno 5] Input/output error
,
Jan 30 2017
,
Feb 1 2017
This is a failure in a swarming shard: https://chromium-swarm.appspot.com/task?id=338fff4adec21c10&refresh=10&show_raw=1 (Milo link: https://luci-milo.appspot.com/buildbot/chromium.fyi/ClangToTMacASan%20tester/3746 ) But this bot has been running fine lately: https://chromium-swarm.appspot.com/bot?id=vm564-m4&sort_stats=total%3Adesc You sure it's not because of this? [75414:7175:0201/112236.093285:24224189386018:ERROR:render_frame_impl.cc(517)] Intentionally crashing (with null pointer dereference) because user navigated to chrome://crash/ The "file handle" that asan_symbolize.py is writing to is not a file, it is a file descriptor to a forked process: https://cs.chromium.org/chromium/src/tools/valgrind/asan/third_party/asan_symbolize.py?q=asan_symbolize.py&sq=package:chromium&dr=C&l=189 If the forked process dies then the write would definitely fail.
,
Feb 1 2017
Removing trooper label because it looks more like a broken test, rather than a trooper issue (Disk space was checked, there are no disk space issues).
,
Feb 2 2017
mac triage: nick@, can you take a look? You touched this test a few months ago :\
,
Feb 2 2017
So this test is exercising crash recovery: it intentionally crashes a renderer process by navigating to the "chrome://crash" debug URL, which results in a call to CrashIntentionally here: https://cs.chromium.org/chromium/src/content/renderer/render_frame_impl.cc?type=cs&l=519. It looks like the problem is that there's a python script /b/swarm_slave/w/irhrDifT/tools/valgrind/asan/asan_symbolize.py that's raising an exception during the w.write, which brings down the whole test? To me it looks like the problem is just that the symbolize script dies while processing a normally crashed process, and the process crash itself is not an error that needs to be fixed.
,
Feb 2 2017
,
Feb 6 2017
+dynamic-tools people for asan_symbolize.py failures. It looks like the ASan crash report is being mangled somewhere between the asan printing and the symbolization script. This could happen if the asan runtime is printing a corrupt report, or if there is some other chrome thread concurrently writing to stderr, which seems more likely.
,
Feb 6 2017
I don't know anything about that script. I'll mark this for infra triage.
,
Feb 7 2017
asan_symbolize.py is normally able to parse the output of a test that crashed intentionally. Something is wrong with the UnbufferedLineConverter() class (https://chromium.googlesource.com/chromium/src/+blame/aa8ec6485091dda9862eb62960b881a60ef9e132/tools/valgrind/asan/third_party/asan_symbolize.py) However it's already wrong that we're using that class. I think we should be symbolizing with LLVMSymbolizer(), not DarwinSymbolizer(). Is LLVM_SYMBOLIZER_PATH set properly?
,
Feb 8 2017
,
Mar 10 2017
clang range: 291275:291321 (via https://luci-milo.appspot.com/buildbot/chromium.fyi/ClangToTMacASan/8174)
,
Mar 10 2017
In that range: https://reviews.llvm.org/D27390 https://reviews.llvm.org/D27378 I wonder if we just need to merge that second patch?
,
Mar 10 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/75865e436efc89aaf076ab43b524782b70e6bd9e commit 75865e436efc89aaf076ab43b524782b70e6bd9e Author: thakis <thakis@chromium.org> Date: Fri Mar 10 22:41:23 2017 Roll tools/valgrind/asan/third_party/asan_symbolize.py 227327:294450 The ASan runtime recently changed to print slice information in stacks (https://reviews.llvm.org/D27390). The last clang roll included that change. The upstream asan_symbolize script was changed to be able to understand slice information in https://reviews.llvm.org/rL291280. We need to merge in that change to make symbolizing asan stacks work again on Mac. This roll brings in the following changes: https://reviews.llvm.org/rL294450 https://reviews.llvm.org/rL294303 https://reviews.llvm.org/rL291280 https://reviews.llvm.org/rL256578 https://reviews.llvm.org/rL251356 https://reviews.llvm.org/rL248962 https://reviews.llvm.org/rL247642 https://reviews.llvm.org/rL231492 BUG= 686113 ,699062 Review-Url: https://codereview.chromium.org/2739213005 Cr-Commit-Position: refs/heads/master@{#456195} [modify] https://crrev.com/75865e436efc89aaf076ab43b524782b70e6bd9e/tools/valgrind/asan/third_party/asan_symbolize.py
,
Mar 10 2017
This should hopefully cycle green; filed bug 700543 for follow-up stuff I saw while debugging.
,
Mar 11 2017
https://build.chromium.org/p/chromium.fyi/builders/ClangToTMacASan%20tester/builds/3962 |
||||||||||
►
Sign in to add a comment |
||||||||||
Comment 1 by spqc...@chromium.org
, Jan 27 2017