Automatically applying components based on crash stacktrace and information from OWNERS files.

If this is incorrect, please apply the Test-Predator-Wrong-Components label.

Automatically assigning owner based on suspected regression changelist https://chromium.googlesource.com/chromium/src/+/d5f07c6849faa9a88e6358eb001de5a75443d791 (MSE: When buffering ByPts, handle keyframe with PTS < previous keyframe).

If this is incorrect, please let us know why and apply the Test-Predator-Wrong-CLs label. If you aren't the correct owner for this issue, please unassign yourself as soon as possible so it can be re-triaged.

I have a local repro, but only with fuzzer -runs option > 1, or alternatively, in a debug asan build with this same case.

Investigating the debug case to see if there's something obvious to fix here.

Hmm. We previously capped the number of trun.sample_count based on GetDemuxerMemoryLimit().

The repro case, however, contains multiple trun.sample_count parses that sum to just under half that cap. However, the number of MP4StreamParser::EnqueueFrame's StreamParserBuffer::CopyFrom() calls exceeds the sum of the trun.sample_counts parsed so far (from extra logs added locally).

Investigating to see why more samples are enqueued than are counted in trun.sample_count.

@#4 oops. The sum of trun.sample_count was >> the # of enqueued (copyfrom()) calls (I misread the logs for trun.sample_count and was off by a factor of 10 in #4).

+mmoroz@: It appears from local repro that the ASAN instrumentation may be introducing *significant* memory overhead/fragmentation:
debug asan repro:
==62159== ERROR: libFuzzer: out-of-memory (used: 2055Mb; limit: 2048Mb)
   To change the out-of-memory limit use -rss_limit_mb=<N>

Live Heap Allocations: 194185709 bytes in 1271178 chunks; quarantined: 8686128 bytes in 46 chunks; 1439 other chunks; total chunks: 1272663; showing top 95% (at most 8 unique contexts)

release build asan repro is similar (though requires more -runs option > 1).

mmoroz@: 2 questions:
1) Why is 194185709 << 2GB, yet 2GB rss limit exceeded?
2) Does ASAN not release all memory free'd within each run? I'm having difficulty finding why -runs > 1 on release mode exceeds the cap, but -runs 1 doesn't.

It's funny, but I'm having some trouble reproducing that locally. I' getting timeout of 67-70 seconds and 1047MB in use with Debug ASan build.

As for the memory usage, 194,185,709 and 8,686,128 is indeed only ~200 MB.

>> 1) Why is 194185709 << 2GB, yet 2GB rss limit exceeded?

I think something else is eating all the memory? How large are all libraries linked in? 

>> 2) Does ASAN not release all memory free'd within each run? I'm having difficulty finding why -runs > 1 on release mode exceeds the cap, but -runs 1 doesn't.

I don't think that ASan keeps anything except of the quarantine, which is fairly small (see ASAN_OPTIONS=...:quarantine_size_mb=10:...). It might be either libFuzzer using some extra memory between runs (e.g. to store coverage info, some tokens extracted from the input, etc) or a memory leak, though it would be reporter I think.

Can you try giving it more memory (e.g. -rss_limit_mb=8192) and trying different number of runs? Will it grow linearly on each execution? To see the final usage append -print_final_stats=1 argument. 

> How large are all libraries linked in? 

I'm not sure how to measure this, especially since they're .so. The list is fairly long. See next:

> Can you try giving it more memory (e.g. -rss_limit_mb=8192) and trying different number of runs? Will it grow linearly on each execution? To see the final usage append -print_final_stats=1 argument. 

Yes. This case (in release mode) doesn't hit 8192MB limit. Watching it in top, RES bounces between (roughly) ~0.750MB and ~2.057GB, dropping down at the beginning of each run. This seems to indicate < ~0.75GB library/static overhead. This appears to indicate large allocation overhead, no leaks, and no significant linear increase in memory utilization across runs.

With -runs=20 -rss_limit_mb=8192 -print_final_stats=1 (and all the other ASAN_OPTIONS from the report):
...
stat::number_of_executed_units: 40  [[[ Double # of runs ]]]
stat::average_exec_per_sec:     0
stat::new_units_added:          0
stat::slowest_unit_time_sec:    0
stat::peak_rss_mb:              2110

@#9, in debug build (still running), the stats (in top) roughly seem to be 75MB higher than release build.

Hmm, ASan's overhead can be huge if there are many small allocations. Can it be the case here?

@#10: final debug build stats:
stat::number_of_executed_units: 40 [[[ Double # of runs ]]]
stat::average_exec_per_sec:     0
stat::new_units_added:          0
stat::slowest_unit_time_sec:    0
stat::peak_rss_mb:              2204

@#11: In this case, 58% of the allocations are indeed small [from debug build repro, not including those outside the 95% of allocations the report lists]

58585264 byte(s) (31%) in 430774 allocation(s) --> 136 bytes/allocation
  (These are StreamParserBuffer instances)

38378208 byte(s) (20%) in 1 allocation(s) --> 37MB/allocation
  (This is a vector of a large number (~1.6 million) of SampleInfo structs)

31015728 byte(s) (16%) in 430774 allocation(s) --> 72 bytes/allocation
  (These are the |data_| compressed bytes in the parent class of each the StreamParserBuffer instances, above)

25166048 byte(s) (13%) in 1 allocation(s) --> 24MB/allocation
  (From FuzzerDriver)

20676432 byte(s) (11%) in 430759 allocation(s) --> 48 bytes/allocation
  (These are from the keyframe map constructed when buffering the StreamParserBuffers into SourceBufferRangeByDts. Aside: fewer of these than the # of buffers can occur even in 100% keyframe streams like audio used in this case, when they overlap enough for splice prerolling)

4720568 byte(s) (2%) in 1 allocation(s) --> 4.5MB/allocation

@#12:
>  Aside: fewer of ...
Likely splicing didn't even occur; the cap being exceeded may have been detected before completion of buffering the last few parsed and constructed streamparserbuffers.

ClusterFuzz testcase 5694492330688512 is flaky and no longer crashes, so closing issue.

If this is incorrect, please add ClusterFuzz-Wrong label and re-open the issue.