New issue
Advanced search Search tips

Issue 918509 link

Starred by 3 users

Issue metadata

Status: Started
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 1
Type: Bug



Sign in to add a comment

CHECK failure: false in aligned_memory.cc

Project Member Reported by ClusterFuzz, Jan 2

Issue description

Detailed report: https://clusterfuzz.com/testcase?key=4934769680580608

Fuzzer: libFuzzer_media_pipeline_integration_fuzzer
Fuzz target binary: media_pipeline_integration_fuzzer
Job Type: x86_libfuzzer_chrome_asan
Platform Id: linux

Crash Type: CHECK failure
Crash Address: 
Crash State:
  false in aligned_memory.cc
  base::AlignedAlloc
  media::VideoFrame::AllocateMemory
  
Sanitizer: address (ASAN)

Reproducer Testcase: https://clusterfuzz.com/download?testcase_id=4934769680580608

Issue filed automatically.

See https://github.com/google/clusterfuzz-tools for instructions to reproduce this bug locally.

Note: This crash might not be reproducible with the provided testcase. That said, for the past 14 days we've been seeing this crash frequently. If you are unable to reproduce this, please try a speculative fix based on the crash stacktrace in the report. The fix can be verified by looking at the crash statistics in the report, a day after the fix is deployed. We will auto-close the bug if the crash is not seen for 14 days.
 
Project Member

Comment 1 by ClusterFuzz, Jan 2

Components: Internals>Core Internals>Media
Labels: Test-Predator-Auto-Components
Automatically applying components based on crash stacktrace and information from OWNERS files.

If this is incorrect, please apply the Test-Predator-Wrong-Components label.
Project Member

Comment 2 by ClusterFuzz, Jan 2

Cc: xhw...@chromium.org
Labels: ClusterFuzz-Auto-CC
Automatically adding ccs based on OWNERS file / target commit history.

If this is incorrect, please add ClusterFuzz-Wrong label.
Cc: kkaluri@chromium.org
Labels: Test-Predator-Wrong
Owner: acourbot@chromium.org
Status: Assigned (was: Untriaged)
Predator and CL could not provide any possible suspects.

Using Code Search for the file, "video_frame.cc" suspecting the below Cl might have caused this issue

Suspect CL: https://chromium.googlesource.com/chromium/src/+/5749f28d663fe9f72392661ac0e31a4ead4fc19a

acourbot@ -- Could you please check whether this is caused with respect to your change, if not please help us in assigning it to the right owner.

Thanks!
Sorry for taking so long to reply, I was OOO. I took a look at the CL, and AFAICT for ffmpeg buffers it really should be a no-op.

However looking at https://clusterfuzz.com/testcase-detail/4934769680580608 I see that the log contains some 6000 lines of the following error:

Stack trace:
../../media/test/pipeline_integration_test_base.cc:358: Failure
Actual function call count doesn't match EXPECT_CALL(*this, OnBufferingStateChange(BUFFERING_HAVE_ENOUGH))...
         Expected: to be called once
           Actual: never called - unsatisfied and active

Could it be related to that failure?
Cc: dalecur...@chromium.org
Also the CL in question has been merged on Nov. 15th, but the statistics show that the failure started on Dec. 20th. Shouldn't the failures have started earlier if that CL introduced it?

+dalecurtis@ in CC for commenting since I am not very knowledgeable about ffmpeg.
Cc: wolenetz@chromium.org
c#4, that's just log spam -- which I thought wolenetz had fixed?

The fuzzer should be spitting out a log with that crash:
https://chromium.googlesource.com/chromium/src/+/c347034892107011aaed034345ef85148afc7cd0/base/memory/aligned_memory.cc#38

So if you run it locally (may have to comment out Env block in fuzzer) it should say what's wrong there.
Fuzzer may have just found the issue, so it may not even be related to your cl.
Thanks Dale - so who owns the next action on this bug?
You should investigate and see if you're able to reproduce locally and if it's actually related to your change. Were you able to reproduce?
@#6, I made targeted suppressions of known spammy logs. New spam like this will need new suppression. If we start getting a huge variety of spam log message sources, we can consider going back to a general close-fd-mask suppression for the fuzzer, but that general suppression wouldn't suppress the logs on local runs (unless care was taken when running locally...)

I'll look into making a targeted suppression fix for this OnBufferingStateChange spam from SRC= fuzzing (but such won't fix the underlying ASAN fault tracked by this bug.)
#@10,  bug 922188  tracks adding this specific log spam suppression (not general suppression..)

Comment 12 by acourbot@chromium.org, Yesterday (45 hours ago)

So I tried reproducing this using the instructions on https://github.com/google/clusterfuzz-tools (clusterfuzz reproduce -f 4934769680580608), but this fails with the following:

ninja: Entering directory `/usr/local/google/home/acourbot/Work/chrome/src/out/clusterfuzz_4934769680580608'
ninja: error: unknown target 'pipeline_integration_fuzzer'

CommandFailedError: `ninja -w 'dupbuild=err' -C /usr/local/google/home/acourbot/Work/chrome/src/out/clusterfuzz_4934769680580608 -j 2400 -l 96 pipeline_integration_fuzzer` failed with the return code 1.

So I tried to build it myself after https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/getting_started.md, to no avail. Do we have other instructions that could help me reproduce this locally?

Comment 13 by acourbot@chromium.org, Yesterday (44 hours ago)

Let's try to understand why the pointer is NULL in the first place. According to http://man7.org/linux/man-pages/man3/posix_memalign.3.html, this could be due to two things:

1) Requested size is 0, in which case posix_memalign() is allowed to store a NULL pointer,
2) posix_memalign() returned an error code, and we set the pointer to NULL ourselves.

We do have a DCHECK() at the beginning of the function to check for size zero, but apparently the fuzz binary is compiled with is_debug = false. So it could be one or the other.

To better understand which one it is, can we merge crrev.com/1424108? It should tell us more precisely what the problem is.



Comment 14 by acourbot@chromium.org, Yesterday (44 hours ago)

Status: Started (was: Assigned)
Mmm, crrev.com/1424108 will probably not help that much, since the log in question won't be compiled in unless we are in debug mode. :/ Still keeping it since it may be useful for local repro once I manage to get a the fuzz test to build and run.

Comment 15 by acourbot@chromium.org, Yesterday (44 hours ago)

Correct link to the CL is crrev.com/c/1424108. >_<

Comment 16 by acourbot@chromium.org, Yesterday (26 hours ago)

Managed to build media_pipeline_integration_fuzzer by just doing a basic gn gen and then passing the arguments specified on the Clusterfuzz report to gn args. It's running on my workstation, trying to reproduce the error.

Comment 17 by acourbot@chromium.org, Today (2 hours ago)

media_pipeline_integration_fuzzer has been running overnight on my workstation with the following arguments (trying to mimic the report as much as possible):

   -rss_limit_mb=2048 -print_final_stats=1 -max_len=1048576

However I could not reproduce the issue. I will need more guidance as to how I can reproduce (especially how I can fetch the same data as used in the test, and how I can specify the seed to use) if I am to keep looking at this.

Also, have we considered that the issue may simply be that the fuzzer runs out of memory? It seems to be working with datasets that get larger and larger over time, so I would not dismiss the possibility that posix_memalign() simply returns ENOMEM.

Sign in to add a comment