The crash is not reproducible, as there is not any single testcase to trigger that, but the OOM happens quite often during the fuzzing.

I don't see any memory leaks neither on crash stats page:
- https://clusterfuzz.com/v2/crash-stats?block=day&days=7&end=417161&fuzzer=libFuzzer_stylesheet_contents_fuzzer&group=platform&number=count

nor in performance report:
- https://clusterfuzz.com/v2/performance-report/libFuzzer_stylesheet_contents_fuzzer/libfuzzer_chrome_asan/latest

Assigning to Marty as per offline discussion.

@mbarbella is this something you're looking into further?

Yeah, I think I see what's going on. Will put a CL together to fix the fuzzer soon.

Seems a bit more complicated than I initially thought. I'm still poking at this a bit, but feel free to steal/reassign if it seems urgent. I'll pass it along to someone else if I can't figure it out.

Hi mbarbella, are you still working on this? Do you know what is causing this issue? Is it to do with the fuzzer or the style parsing code?

Marty is not looking into this right now, so this is yours Darren! :)

My understanding is that we're OOMing because we hit the RSS limit of 2048MB. I did some googling and found this: https://stackoverflow.com/questions/41613682/llvm-libfuzzer-rss-memory-increase

My observations:
- Quarantine should not be an issue, as quarantine_size_mb appears to be 10.
- All the top allocations listed in the error appear to be from the fuzzer. InputCorpus ::AddToCorpus appears frequently.
- Yet, the allocations aren't large enough to account for the 2GBs.
- Basically every line in the fuzzer output was a 'NEW', whereas locally I saw a good mix of NEW and REDUCE. Not sure if this is normal.
- There was an error message saying:
    INFO: libFuzzer disabled leak detection after every mutation.
    Most likely the target function accumulates allocated 
    memory in a global state w/o actually leaking it.
  So it's quite possible some global state is accumulating memory in stylesheet parsing or the fuzzer. Again, not sure if this is normal.
- Corpus size is only 16MB (assuming that's what "corp: 12762/16Mb") means.
- We are doing a GC collect every iteration.

Random guesses:
- We're maintaining some global state / cache that's accumulating (not detected as a memory leak). I'm not aware of anything like that in the parser, but I'll search around.
- The stylesheet fuzzer is caught in a bad state / local minima due to bad corpus / lack of coverage.

Can anyone with more knowledge about fuzzing comment on this? Just as a way of turning it off and on again...how bad would it be if we cleared the corpus and gave it a better dictionary and seed corpus (currently the seed corpus contains very small snippets of CSS and random png images; I think seeding it with popular CSS frameworks like bootstrap would be better?).

+csharrison for some advice

+1 for seeding with a better corpus. Feel free to make that change.

CollectAllGarbage uses kNoHeapPointersOnStack, even though we have some heap pointers on the stack. I *think* this is OK and those will be collected, but it might make sense to put all the CSS stuff in a separate block so they are out of scope at that point.

We do have global objects that could potentially be "leaking" here. Maybe AtomicString tables? I'm not sure of any way to dump the heap to check but maybe memory / libfuzzer folks have advice.

Hm, ~StringImpl should remove a string from the AtomicString table, so maybe it's not that.

I tried putting the CSS stuff in a separate block but it doesn't seem to help.

Do you know if I can get the CF test cases for all the other OOMs that are happening? The one in the linked report has stuff about the 'repeat' CSS function [1]. I wonder if that's just a coincidence or maybe we are "leaking" in that code. I did a quick experiment by seeding the fuzzer with the CF test case and the RSS seems to steadily increase beyond 1GB, whereas normally the RSS peaks at around 700MB. But again that could just be a coincidence.

[1] https://developer.mozilla.org/en-US/docs/Web/CSS/repeat

mmoroz might know, I tried looking through [1] but couldn't find anything.

In any case, yeah repeat() definitely seems like it could cause high memory usage according to the code [2]. I couldn't see any leakage though. +erikchen is there an easy way to get heap snapshots for libfuzzer runs? We're trying to trace some potential leaks.

[1]: https://clusterfuzz.com/v2/crash-stats?block=day&days=7&end=418632&fuzzer=libFuzzer_stylesheet_contents_fuzzer&group=platform&number=count&sort=total_count

[2]: https://cs.chromium.org/chromium/src/third_party/WebKit/Source/core/style/GridArea.h?rcl=07d92b832e636397f113353dc9413c247996e250&l=48

on a side note, would it be better to check in a bunch of seed CSS files into chromium or just upload it directly to the corpus directory?

...and as an extra data point, if I'm reading [1] correctly, there hasn't been any new OOMs in ASAN builds since August 10. All the OOMs are coming from MSAN? This also coincides with a dip in the total OOMs in [2]. Were there any changes in the fuzzer around that time?

[1] https://clusterfuzz.com/v2/fuzzer-stats/by-job/2017-08-10/2017-10-03/fuzzer/libFuzzer_stylesheet_contents_fuzzer?noredirect=1

[2] https://clusterfuzz.com/v2/crash-stats?block=day&days=150&end=418632&fuzzer=libFuzzer_stylesheet_contents_fuzzer&group=platform&number=count&q=out-of-memory&sort=total_count

> is there an easy way to get heap snapshots for libfuzzer runs? We're trying to trace some potential leaks.

I'm not very familiar with libfuzzer. Assuming libfuzzer isn't dependent on any of the sanitizers [ASAN, MSAN, etc.] it should be feasible to hook up heap profiling. 

But if you're looking to do a one-off heap analysis on something that is runnable locally, the easiest way would be to use ETW on Windows or Instruments on macOS, where things generally "just work" out of the box

Thanks for the info! Unfortunately, these failures are coming from ASAN builds :)

Using ETW or Instruments SGTM, I think this repros on Mac. Do you know of a good tool for heap analysis on Linux? Maybe "perf" could do it, though I've only ever used it for CPU analysis. 

> Thanks for the info! Unfortunately, these failures are coming from ASAN builds :)

Hm, yeah, that can be tricky. I don't know of any good tools for heap analysis on Linux - and I'm not sure if any of the tools on other platforms work with ASAN. Might be worth checking with some of the Lexan folk, who are probably more familiar.

+rnk, who should at least be able to point us in the right direction

We had an intern port libFuzzer and the sanitizer coverage it relies on to Windows, but it bitrotted because we didn't have the bandwidth to maintain the port. dynamic-tools@ also generally isn't willing to spend much effort maintaining Windows ports. I'll make sure to let kcc@ know that we have potential users. He believed there were no users to motivate the maintenance.

I also don't know any good heap profilers for Linux. =/

mmoroz, could [1] be causing this issue, since it looks like only MSAN is crashing from OOM?

[1] https://bugs.chromium.org/p/chromium/issues/detail?id=770430

Sorry for being silent in last 24 hours. I'm reading all the comments above and thinking about that right now.

Adding dictionary is definitely a good idea. Thanks shend@ for uploading the CL: https://chromium-review.googlesource.com/c/chromium/src/+/696441

It's weird that we cannot land it due to flaky tests (I guess?). I clicked "Submit" once again. If it doesn't go through, probably worth filing a bug and landing with CQ bypass.

Darren, your c#16 is correct. We haven't seen OOMs from this fuzzer built with ASan since August ~9-11: https://clusterfuzz.com/v2/fuzzer-stats/by-day/2017-07-01/2017-10-04/fuzzer/libFuzzer_stylesheet_contents_fuzzer/job/libfuzzer_chrome_asan

Check out "peak_mem_mb" and "oom_count" columns. Compare with MSan:

https://clusterfuzz.com/v2/fuzzer-stats/by-day/2017-07-01/2017-10-04/fuzzer/libFuzzer_stylesheet_contents_fuzzer/job/libfuzzer_chrome_msan

I'm not totally sure why it happened. There were some changes in libFuzzer aimed to optimize memory usage. Plus, mbarbella@ might have landed some fixes as he had ideas in the beginning of August.

Anyway, it's good to see this fuzzer working and getting new coverage in the last two days.

As for MSan, it has a bigger memory overhead than ASan. We see lots of OOMs with MSan especially from memory-consuming stuff like media fuzzers. Often we have to WontFix those as memory usage looks reasonable. I'll raise the priority on https://bugs.chromium.org/p/chromium/issues/detail?id=770430, so we'll think a bit more about other potential solutions.

shend@, feel free to close this issue unless you have other ideas how to reduce memory usage :)

Awesome, thanks for the help everyone! I'll close this now and hopefully we'll figure out something for MSan builds.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/745bf24448cfe21218b44a9f8621b08de8a8fc11

commit 745bf24448cfe21218b44a9f8621b08de8a8fc11
Author: Darren Shen <shend@chromium.org>
Date: Thu Oct 05 08:30:10 2017

Add a CSS fuzzer dictionary for stylesheet_contents_fuzzer.

Currently the stylesheet_contents_fuzzer, which fuzzs CSS stylesheet
parsing, has no dictionary, which means most of the inputs are garbage
and will be ignored at the parsing level. This patch adds some common
strings seen in CSS stylesheets, which would hopefully produce more
valid tokens that can expose more bugs at the parsing level.

Bug:  752194 
Change-Id: Ibf596a0a78c5b1bfb042f2c9e952ebdc07790a42
Reviewed-on: https://chromium-review.googlesource.com/696441
Reviewed-by: meade_UTC10 <meade@chromium.org>
Reviewed-by: Max Moroz <mmoroz@chromium.org>
Reviewed-by: Oliver Chang <ochang@chromium.org>
Commit-Queue: Darren Shen <shend@chromium.org>
Cr-Commit-Position: refs/heads/master@{#506677}
[modify] https://crrev.com/745bf24448cfe21218b44a9f8621b08de8a8fc11/third_party/WebKit/Source/core/BUILD.gn
[add] https://crrev.com/745bf24448cfe21218b44a9f8621b08de8a8fc11/third_party/WebKit/Source/core/css/css.dict

Issue 749810 has been merged into this issue.