I wrote a web page that creates web workers one after another: https://ulan.github.io/misc/workers/index.html

I checked that it creates as many CodeRanges as there are workers, but I cannot reproduce the CFG leak reported in 870054 on Windows 10 with Chrome 68.

Bruce, does the example above reproduce the leak on your machine?

After spawning 3000 workers I can see the leak on Chrome 68. Chrome private memory footprint grows from ~50MB to ~200MB.

Chrome Canary stays in 50MB-70MB range.

I think we are ready to turn this into a benchmark.

On desktop we could run:
- https://ulan.github.io/misc/workers/index.html?auto=1&workers=2000

On mobile:
- https://ulan.github.io/misc/workers/index.html?auto=1&workers=1000

Ned, Juan, do you think we can add this to the system health Chrome stories?

This benchmark would have prevented a massive memory leak (see write-ups by Bruce in  crbug.com/870054  and https://randomascii.wordpress.com/2018/08/16/24-core-cpu-and-i-cant-type-an-email-part-one/)

This is also useful for tracking web worker startup performance as it runs many web workers.

Thanks for the test page. Yes, I can definitely see the leak. I ran my VirtualScan tool and it showed the number of CFG blocks smoothly increasing (the CFG memory and page-table counts also increased, but CFG block count is the most sensitive).

By the end the results looked like this:

     Scan time, Committed, page tables, committed blocks
Total:  4.165s, 274.4 MiB,     8.3 MiB,   4368, 65 code blocks, in process 26536
  CFG:  4.161s,  50.8 MiB,     7.2 MiB,   3561

That's a leak of ~25 MiB of committed CFG memory, 7.1 MiB of CFG page tables, and 3,500 CFG blocks. Also notice that by the end it was taking 4.161 s to scan the CFG region, and during that time the page would hang!

So yes, this reproduces all aspects of the bug. If you just look at private commit then it will be less obvious, but still there.

After running the page again (2,000 workers) I get these results - getting linearly worse:

     Scan time, Committed, page tables, committed blocks
Total:  7.988s, 324.8 MiB,    14.2 MiB,   7918, 68 code blocks, in process 26536
  CFG:  7.985s,  94.2 MiB,    13.1 MiB,   7057

I also tested on beta and canary and confirmed that the bug is fixed there. The number of CFG blocks stays steady at 64.

I think I spot a very small hang when doing the scanning. The tool claims that the scanning takes ~100 ms which is just barely detectable. That micro-hang is reduced by Microsoft in more recent versions of Windows and there is nothing else we can do about it. Here are the canary test results:

     Scan time, Committed, page tables, committed blocks
Total:  0.103s, 250.9 MiB,     1.2 MiB,    857, 65 code blocks, in process 15696
  CFG:  0.099s,  28.8 MiB,     0.1 MiB,     64

For regression testing there are clearly advantages to monitoring the number of CFG memory chunks rather than the total committed memory. It turns out that doing this in-process is quite straightforward. My VirtualStress program scans its own address space to find the CFG allocation (it assumes it is a 64-bit process). It's about twenty lines of code:

https://github.com/randomascii/blogstuff/blob/master/cfg/VAllocStress/VAllocStress.cpp#L80

I just turned that into a function that counts the committed CFG blocks in a 64-bit process:

size_t NumCFGBlocks()
{
  // Count the number of blocks in the current reservation.
  size_t block_count = 0;
  // Find our own CFG reservation - just scan the address space looking for a 2 TB
  // allocation.
  const void* cur_base = nullptr;
  for (const char* p = nullptr; /**/; /**/)
  {
    MEMORY_BASIC_INFORMATION info = {};
    SIZE_T result = VirtualQuery(p, &info, sizeof(info));
    // If no CFG region is found then eventually VirtualQuery will fail and the
    // loop will exit here.
    if (result != sizeof(info))
      break;
    if (info.AllocationBase != cur_base)
    {
      // Presumably only the cfg block will be 2 TiB.
      ptrdiff_t cur_base_size = p - static_cast<const char*>(cur_base);
      if (cur_base_size == 0x20000000000)
      {
        // We found it!
        return block_count;
      }
      cur_base = info.AllocationBase;
      block_count = 0;
    }
    p += info.RegionSize;
    if (info.State == MEM_COMMIT)
      ++block_count;
  }
  return 0;
}

We don't want to add a synthetic story to system health. But you can add that one to memory.desktop (erikchen@ is owner, iirc)

Yup, that seems like a reasonable benchmark to add to memory.desktop

Obviously the CFG block counting only detects one specific issue and only on Windows, so monitoring committed memory (and creating enough worker threads to make leaks visible) would be great as a cross-platform memory.desktop benchmark.

Bruce, thanks for testing!

Adding to memory.desktop sounds good. I uploaded: https://chromium-review.googlesource.com/c/chromium/src/+/1202084

To keep the run time reasonable, I reduced the number of workers (to 1000) and the delay between chunk allocations in each worker (10ms => 1ms). The benchmark now requires about 55 seconds to run.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/790bfc34142684aa0689a0cb8a84219dad133d06

commit 790bfc34142684aa0689a0cb8a84219dad133d06
Author: Ulan Degenbaev <ulan@chromium.org>
Date: Tue Sep 04 16:17:28 2018

Add memory.desktop/WebWorker benchmark.

It is useful for catching memory leaks related to web worker creation.

Bug: 875772
Change-Id: I5fda4e8cc78e62fd7e3420b4aed02b8219499983
Reviewed-on: https://chromium-review.googlesource.com/1202084
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Erik Chen <erikchen@chromium.org>
Commit-Queue: Ulan Degenbaev <ulan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#588533}
[modify] https://crrev.com/790bfc34142684aa0689a0cb8a84219dad133d06/tools/perf/benchmarks/memory.py
[add] https://crrev.com/790bfc34142684aa0689a0cb8a84219dad133d06/tools/perf/page_sets/web_worker_stories.py
[add] https://crrev.com/790bfc34142684aa0689a0cb8a84219dad133d06/tools/perf/page_sets/web_workers/index.html
[add] https://crrev.com/790bfc34142684aa0689a0cb8a84219dad133d06/tools/perf/page_sets/web_workers/worker.js

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/790bfc34142684aa0689a0cb8a84219dad133d06

commit 790bfc34142684aa0689a0cb8a84219dad133d06
Author: Ulan Degenbaev <ulan@chromium.org>
Date: Tue Sep 04 16:17:28 2018

Add memory.desktop/WebWorker benchmark.

It is useful for catching memory leaks related to web worker creation.

Bug: 875772
Change-Id: I5fda4e8cc78e62fd7e3420b4aed02b8219499983
Reviewed-on: https://chromium-review.googlesource.com/1202084
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Erik Chen <erikchen@chromium.org>
Commit-Queue: Ulan Degenbaev <ulan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#588533}
[modify] https://crrev.com/790bfc34142684aa0689a0cb8a84219dad133d06/tools/perf/benchmarks/memory.py
[add] https://crrev.com/790bfc34142684aa0689a0cb8a84219dad133d06/tools/perf/page_sets/web_worker_stories.py
[add] https://crrev.com/790bfc34142684aa0689a0cb8a84219dad133d06/tools/perf/page_sets/web_workers/index.html
[add] https://crrev.com/790bfc34142684aa0689a0cb8a84219dad133d06/tools/perf/page_sets/web_workers/worker.js

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/790bfc34142684aa0689a0cb8a84219dad133d06

commit 790bfc34142684aa0689a0cb8a84219dad133d06
Author: Ulan Degenbaev <ulan@chromium.org>
Date: Tue Sep 04 16:17:28 2018

Add memory.desktop/WebWorker benchmark.

It is useful for catching memory leaks related to web worker creation.

Bug: 875772
Change-Id: I5fda4e8cc78e62fd7e3420b4aed02b8219499983
Reviewed-on: https://chromium-review.googlesource.com/1202084
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Erik Chen <erikchen@chromium.org>
Commit-Queue: Ulan Degenbaev <ulan@chromium.org>
Cr-Commit-Position: refs/heads/master@{#588533}
[modify] https://crrev.com/790bfc34142684aa0689a0cb8a84219dad133d06/tools/perf/benchmarks/memory.py
[add] https://crrev.com/790bfc34142684aa0689a0cb8a84219dad133d06/tools/perf/page_sets/web_worker_stories.py
[add] https://crrev.com/790bfc34142684aa0689a0cb8a84219dad133d06/tools/perf/page_sets/web_workers/index.html
[add] https://crrev.com/790bfc34142684aa0689a0cb8a84219dad133d06/tools/perf/page_sets/web_workers/worker.js