Bots: fuzz targets can hang in a way that libFuzzer will not detect timeout |
|||
Issue descriptionNoticed that on a code coverage bot: mmoroz@code-coverage-linux-0001:~$ cat ../coverage-bot/scripts/_bot.log | egrep "^### Start |^### End" ### Start /home/coverage-bot/scripts/code_coverage_loop.bash at Fri Jun 15 01:00:03 UTC 2018 ### Start /home/coverage-bot/scripts/build_targets.bash at Fri Jun 15 01:02:29 UTC 2018 ### End /home/coverage-bot/scripts/build_targets.bash at Fri Jun 15 05:32:16 UTC 2018 ### Start /home/coverage-bot/scripts/run_test_targets.bash at Fri Jun 15 05:32:16 UTC 2018 ### End /home/coverage-bot/scripts/run_test_targets.bash at Fri Jun 15 09:39:44 UTC 2018 ### Start /home/coverage-bot/scripts/run_fuzz_targets.bash at Fri Jun 15 09:39:44 UTC 2018 mmoroz@code-coverage-linux-0001:~$ date Mon Jun 18 03:29:45 UTC 2018 mmoroz@code-coverage-linux-0001:~$ tail -n 30 ../coverage-bot/scripts/_bot.log ++ wc -l + n_child_proc=13 + '[' 13 -eq 16 ']' + for target in '$(cat $BUILT_FUZZ_TARGETS_LOG)' ++ basename zucchini_raw_gen_fuzzer + target=zucchini_raw_gen_fuzzer + echo -e '\n\nRunning fuzz target: zucchini_raw_gen_fuzzer' Running fuzz target: zucchini_raw_gen_fuzzer + sleep 2 + /home/coverage-bot/scripts/run_single_fuzz_target.bash /chromium/src/out/coverage zucchini_raw_gen_fuzzer /chromium/src/fuzzer_stats ++ jobs -p ++ wc -l + n_child_proc=13 + '[' 13 -eq 16 ']' + for target in '$(cat $BUILT_FUZZ_TARGETS_LOG)' ++ basename zucchini_ztf_gen_fuzzer + target=zucchini_ztf_gen_fuzzer + echo -e '\n\nRunning fuzz target: zucchini_ztf_gen_fuzzer' Running fuzz target: zucchini_ztf_gen_fuzzer + sleep 2 + /home/coverage-bot/scripts/run_single_fuzz_target.bash /chromium/src/out/coverage zucchini_ztf_gen_fuzzer /chromium/src/fuzzer_stats ++ jobs -p ++ wc -l + n_child_proc=14 + '[' 14 -eq 16 ']' + wait mmoroz@code-coverage-linux-0001:~$ ps aux | egrep _fuzzer mmoroz 24266 0.0 0.0 14796 1016 pts/0 S+ 03:30 0:00 grep -E --color=auto _fuzzer coverag+ 25196 0.0 0.0 13112 3084 ? S Jun15 0:00 /bin/bash -ux /home/coverage-bot/scripts/run_single_fuzz_target.bash /chromium/src/out/coverage content_security_policy_fuzzer /chromium/src/fuzzer_stats coverag+ 26124 0.1 0.1 1327368 233156 ? Sl Jun15 6:09 /chromium/src/out/coverage/content_security_policy_fuzzer -runs=0 -timeout=60 corpus/content_security_policy_fuzzer Once I killed that proccess, the bot was unblocked.
,
Jun 18 2018
I guess we have to use timeout in a way similar to how Abhishek added that for running test targets: https://chrome-internal.googlesource.com/chrome/tools/code-coverage/+/master/scripts/run_test_targets.bash#19
,
Jun 18 2018
Actually, this issue is not specific for a single fuzz target: mmoroz@code-coverage-linux-0004:~$ ps aux | egrep _fuzzer coverag+ 4181 0.4 0.2 6708404 253676 ? Sl Jun15 16:04 /chromium/src/out/coverage/pdfium_fuzzer -runs=0 -timeout=60 corpus/pdfium_fuzzer mmoroz 30028 0.0 0.0 14796 964 pts/0 S+ 03:35 0:00 grep -E --color=auto _fuzzer coverag+ 31520 0.0 0.0 13112 2964 ? S Jun15 0:00 /bin/bash -ux /home/coverage-bot/scripts/run_single_fuzz_target.bash /chromium/src/out/coverage pdfium_fuzzer /chromium/src/fuzzer_stats mmoroz@code-coverage-linux-0004:~$ tail -f ../coverage-bot/scripts/_bot.log Running fuzz target: zucchini_ztf_gen_fuzzer + sleep 2 + /home/coverage-bot/scripts/run_single_fuzz_target.bash /chromium/src/out/coverage zucchini_ztf_gen_fuzzer /chromium/src/fuzzer_stats ++ wc -l ++ jobs -p + n_child_proc=15 + '[' 15 -eq 16 ']' + wait ^C mmoroz@code-coverage-linux-0004:~$ sudo kill 4181 31520 mmoroz@code-coverage-linux-0004:~$ tail -f ../coverage-bot/scripts/_bot.log + RUN_RETRIES=3 + TARGET=layout_tests + TIMEOUT=5h + [[ '' == \1 ]] + echo 'Running layout tests' Running layout tests ++ seq 0 3 + for i in '$(seq 0 $RUN_RETRIES)' + [[ 0 -gt 0 ]] + run_layout_tests ^C I can wrap running fuzz targets with a timeout tomorrow.
,
Jun 18 2018
+1
,
Jun 18 2018
The following revision refers to this bug: https://chrome-internal.googlesource.com/chrome/tools/code-coverage/+/923259efc77cbb969c1b745f93a65f9e322cba12 commit 923259efc77cbb969c1b745f93a65f9e322cba12 Author: Max Moroz <mmoroz@google.com> Date: Mon Jun 18 16:15:18 2018
,
Jun 23 2018
|
|||
►
Sign in to add a comment |
|||
Comment 1 by mmoroz@chromium.org
, Jun 18 2018Labels: -Pri-3 Coverage-v2-Blocker Pri-1