New issue
Advanced search Search tips

Issue 846721 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: Jun 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

minidump_fuzzer is too noisy

Project Member Reported by mmoroz@chromium.org, May 25 2018

Issue description

For example, it's log from code coverage bot[1] is 34M large, even though it simply runs over 1771 corpus units. That amount of logging creates unnecessary overhead and complicates interaction with fuzzing infrastructure.


[1]: https://chromium-coverage.appspot.com/reports/560197/linux/metadata/index.html
 

Comment 1 by mmoroz@chromium.org, May 25 2018

Owner: mmoroz@chromium.org
Status: Started (was: Untriaged)
I have one solution in mind, let me upload it.
Project Member

Comment 3 by bugdroid1@chromium.org, May 29 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/breakpad/breakpad/+/27c001192ba9b78fe8f47c5a51a2d0a371860f12

commit 27c001192ba9b78fe8f47c5a51a2d0a371860f12
Author: Max Moroz <mmoroz@chromium.org>
Date: Tue May 29 19:24:32 2018

Add SEVERITY_CRITICAL to logging, that would help to suppress logs from the fuzz target.

Follow-up CL will be https://chromium-review.googlesource.com/c/chromium/src/+/1073395

Bug:  846721 
Change-Id: Ie9e6dc5c5ef6b035c414fbdc4f711f995b52f4d7
Reviewed-on: https://chromium-review.googlesource.com/1073394
Reviewed-by: Will Harris <wfh@chromium.org>

[modify] https://crrev.com/27c001192ba9b78fe8f47c5a51a2d0a371860f12/src/processor/logging.cc
[modify] https://crrev.com/27c001192ba9b78fe8f47c5a51a2d0a371860f12/src/processor/logging.h

Project Member

Comment 4 by bugdroid1@chromium.org, May 29 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/946a8b8861cbe157bfa45c438bec066dbff85ee2

commit 946a8b8861cbe157bfa45c438bec066dbff85ee2
Author: Max Moroz <mmoroz@chromium.org>
Date: Tue May 29 21:02:01 2018

Roll src/third_party/breakpad/breakpad/ c83fcf647..27c001192 (2 commits).

https://chromium.googlesource.com/breakpad/breakpad.git/+log/c83fcf647..27c001192

$ git log c83fcf647..27c001192 --date=short --no-merges --format='%ad %ae %s'
2018-05-27 mmoroz@chromium.org Add SEVERITY_CRITICAL to logging, that would help to suppress logs from the fuzz target.
2018-04-13 thestig@chromium.org Read Linux si_code in minidump_processor.

Bug:  846721 
Change-Id: Ib7d34a0dd393951be8ab6e30d0c5f341fed3389b
Reviewed-on: https://chromium-review.googlesource.com/1077078
Reviewed-by: Will Harris <wfh@chromium.org>
Commit-Queue: Max Moroz <mmoroz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#562582}
[modify] https://crrev.com/946a8b8861cbe157bfa45c438bec066dbff85ee2/DEPS

Project Member

Comment 5 by bugdroid1@chromium.org, May 30 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/e7f95160938eab43f47a7f2c4fc76a14eacfe7c0

commit e7f95160938eab43f47a7f2c4fc76a14eacfe7c0
Author: Max Moroz <mmoroz@chromium.org>
Date: Wed May 30 22:21:14 2018

[breakpad] use BPLOG_MINIMUM_SEVERITY=SEVERITY_CRITICAL to suppress minidump_fuzzer logs.

The CL depends on https://chromium-review.googlesource.com/c/breakpad/breakpad/+/1073394

Bug:  846721 
Change-Id: I183e6adf370da3de30f16b2b6917a9a5d927e37b
Reviewed-on: https://chromium-review.googlesource.com/1073395
Reviewed-by: Lei Zhang <thestig@chromium.org>
Reviewed-by: Leonard Grey <lgrey@chromium.org>
Commit-Queue: Max Moroz <mmoroz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#563029}
[modify] https://crrev.com/e7f95160938eab43f47a7f2c4fc76a14eacfe7c0/third_party/breakpad/BUILD.gn

Comment 6 by mmoroz@chromium.org, May 31 2018

Looks like there is something else blowing up the logs: https://chromium-coverage.appspot.com/reports/563089/linux/metadata/minidump_fuzzer.log

Comment 7 by wfh@chromium.org, May 31 2018

Indeed. I added a comment in crrev.com/c/1073395 - what happens when you compile and run it manually?

Comment 8 by mmoroz@chromium.org, May 31 2018

Thanks, Will. You're right, and I've uploaded https://chromium-review.googlesource.com/c/chromium/src/+/1081349 just minutes ago
Project Member

Comment 9 by bugdroid1@chromium.org, Jun 1 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/04f17ba8ede78dda3997d2fcbcc02d7b60b60788

commit 04f17ba8ede78dda3997d2fcbcc02d7b60b60788
Author: Max Moroz <mmoroz@chromium.org>
Date: Fri Jun 01 04:02:06 2018

[breakpad] Move BPLOG_MINIMUM_SEVERITY macro definition into configs.

R=thestig@chromium.org

Bug:  846721 
Change-Id: I733ded1abc797b708f486f7f8d048570e9a7df4c
Reviewed-on: https://chromium-review.googlesource.com/1081349
Commit-Queue: Max Moroz <mmoroz@chromium.org>
Reviewed-by: Lei Zhang <thestig@chromium.org>
Cr-Commit-Position: refs/heads/master@{#563527}
[modify] https://crrev.com/04f17ba8ede78dda3997d2fcbcc02d7b60b60788/third_party/breakpad/BUILD.gn

And that also is not enough... 
13:59:07:322853.log
371 KB View Download
Cc: thestig@chromium.org
I'm not even sure where those lines come from. Valgrind? https://cs.chromium.org/search/?q=%22Suppressions+used%22&sq=package:chromium&type=cs

Comment 12 by wfh@chromium.org, Jun 1 2018

is it not possible to run this locally and try and eliminate the noise rather than relying on having to run this on the infrastructure? it seems that would give a far lower turn-around on fixes and more assurance that the log spam is being successfully suppressed?
That's a good point, but I haven't seen those messages when tested my changes locally. Even the first CL was enough locally. Probably I'm doing something wrong :)
Where did the log in comment 10 come from? Valgrind is not in use any more. The Valgrind suppression output did not have the "bytes" field.
The log in c#10 is from ClusterFuzz: https://pantheon.corp.google.com/storage/browser/clusterfuzz-libfuzzer-logs/libFuzzer_minidump_fuzzer/libfuzzer_chrome_asan/

Turns out, it's being printed by LSAN: https://github.com/llvm-mirror/compiler-rt/blob/master/lib/lsan/lsan_common.cc#L510

So, this is a CF side issue, I'll file a separate bug for that.

However, on the coverage bot I'm also seeing the following output:

OS|0xb7b7b7b7|558450945.4802999.3082270647 
CPU|0x6136||56
GPU|||
Crash|No crash||

OS|0xb7b7b7b7|0.3082270647.3082270647 
CPU|x86|贩贩贩贩 family 0 model 1 stepping 0|0
GPU|||
Crash|No crash||



e.g. https://chromium-coverage.appspot.com/reports/563766/linux/metadata/minidump_fuzzer.log
Looks like stackwalk_common is using printf() instead of any logging functions: https://cs.chromium.org/chromium/src/third_party/breakpad/breakpad/src/processor/stackwalk_common.cc?type=cs&q=%22no+crash%22+file:%5Esrc/third_party/breakpad/+package:%5Echromium$&g=0&l=787 and https://cs.chromium.org/chromium/src/third_party/breakpad/breakpad/src/processor/stackwalk_common.cc?type=cs&q=%22no+crash%22+file:%5Esrc/third_party/breakpad/+package:%5Echromium$&g=0&l=878

That is being called directly from the fuzz target for valid minidumps: https://cs.chromium.org/chromium/src/third_party/breakpad/minidump_fuzzer.cc?type=cs&g=0&l=83

And it makes sense, I guess, since there is also some interesting code inside that function.

With that, I don't see any other solution rather than adding close_fd_mask=1 flag to libfuzzer_options.

The coverage bot will not use that flag, but it would be acceptable, as the log right now is less than 300KB: https://chromium-coverage.appspot.com/reports/563766/linux/metadata/index.html
Project Member

Comment 18 by bugdroid1@chromium.org, Jun 4 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/clusterfuzz/+/faffe078419854defaff271a5891a45fe55c183b

commit faffe078419854defaff271a5891a45fe55c183b
Author: Max Moroz <mmoroz@google.com>
Date: Mon Jun 04 04:37:14 2018

Project Member

Comment 19 by bugdroid1@chromium.org, Jun 4 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ea3d7503de2dd3aa0a605f58dde02810b73ee1a5

commit ea3d7503de2dd3aa0a605f58dde02810b73ee1a5
Author: Max Moroz <mmoroz@chromium.org>
Date: Mon Jun 04 22:24:47 2018

[breakpad] minidump_fuzzer: use close_fd_mask=1 instead of 3, add a comment to clarify.

Bug:  846721 
Change-Id: I68fdd54ba264d40056b60aa25dd0c624b25648c3
Reviewed-on: https://chromium-review.googlesource.com/1084195
Reviewed-by: Lei Zhang <thestig@chromium.org>
Commit-Queue: Max Moroz <mmoroz@chromium.org>
Cr-Commit-Position: refs/heads/master@{#564269}
[modify] https://crrev.com/ea3d7503de2dd3aa0a605f58dde02810b73ee1a5/third_party/breakpad/BUILD.gn

Status: Fixed (was: Started)

Sign in to add a comment