New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 660980 link

Starred by 2 users

Issue metadata

Status: Fixed
Owner:
Closed: May 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 2
Type: ----



Sign in to add a comment

Courgette: AssemblyProgram Reduction

Project Member Reported by hua...@chromium.org, Oct 31 2016

Issue description

This is a tracking bug for Courgette memory reduction related to AssemblyProgram

Details: go/courgette-reduce-assembly-program

Phase 1: Change how instructions storage format in AssemblyProgram.
Phase 2: Remove instruction storage by streaming data directly to EncodedProgram.

 
Project Member

Comment 2 by bugdroid1@chromium.org, Nov 9 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/7b221a5ec1175ffba9955aede222b9bdd4056183

commit 7b221a5ec1175ffba9955aede222b9bdd4056183
Author: huangs <huangs@chromium.org>
Date: Wed Nov 09 22:28:23 2016

[Courgette] Refactor: Add and use Instruction*Receptor classes; call ParseFile() in 2 passes.

We wish to reduce AssemblyProgram's memory usage on instruction storage.
This requires changing Disassembler*::Disassemble() to call ParseFile()
in 2 passes: (1) count bytes needed (so we can preallocate) (2) store
bytes. This CL implements the 2-pass system, focusing on callers. Details:
- Add AssemblyProgram::GenerateInstructions() as main entrance point
  from Disassembler*. It takes InstructionGenerator, which is just
  Disassembler*::ParseFile() wrapped in a Closure.
- Change Disassembler*::ParseFile() and downstream functions to const,
  to assert absence of Disassembler state change for the passes.
  - Exception: For ELF, we need to stort abs32_locations_ and
    re32_locations_ to obtain different traversal orders. This doesn't
    add/remove elements. So we make these containers mutable.
- Emit*() functions are moved from AssemblyProgram to new interface
  InstructionReceptor. Updating all Emit*() calls to accommodate.
- Add InstructionReceptor interface with 2 implementations:
  - InstructionCountReceptor for pass 1 (no-op in this CL).
  - InstructionStoreReceptor for pass 2 (calls AssemblyProgram::Emit*()
    in this CL).
- Rename Emit*Instruction() to Emit*().

TEST=Correctness without behavioral change; Courgette-gen and Courgette-apply are not slowed down, and do not use more memory.
BUG= 660980 

Review-Url: https://codereview.chromium.org/2462993003
Cr-Commit-Position: refs/heads/master@{#431053}

[modify] https://crrev.com/7b221a5ec1175ffba9955aede222b9bdd4056183/courgette/adjustment_method_unittest.cc
[modify] https://crrev.com/7b221a5ec1175ffba9955aede222b9bdd4056183/courgette/assembly_program.cc
[modify] https://crrev.com/7b221a5ec1175ffba9955aede222b9bdd4056183/courgette/assembly_program.h
[modify] https://crrev.com/7b221a5ec1175ffba9955aede222b9bdd4056183/courgette/disassembler_elf_32.cc
[modify] https://crrev.com/7b221a5ec1175ffba9955aede222b9bdd4056183/courgette/disassembler_elf_32.h
[modify] https://crrev.com/7b221a5ec1175ffba9955aede222b9bdd4056183/courgette/disassembler_elf_32_arm.cc
[modify] https://crrev.com/7b221a5ec1175ffba9955aede222b9bdd4056183/courgette/disassembler_elf_32_arm.h
[modify] https://crrev.com/7b221a5ec1175ffba9955aede222b9bdd4056183/courgette/disassembler_elf_32_x86.cc
[modify] https://crrev.com/7b221a5ec1175ffba9955aede222b9bdd4056183/courgette/disassembler_elf_32_x86.h
[modify] https://crrev.com/7b221a5ec1175ffba9955aede222b9bdd4056183/courgette/disassembler_win32.cc
[modify] https://crrev.com/7b221a5ec1175ffba9955aede222b9bdd4056183/courgette/disassembler_win32.h
[modify] https://crrev.com/7b221a5ec1175ffba9955aede222b9bdd4056183/courgette/disassembler_win32_x64.cc
[modify] https://crrev.com/7b221a5ec1175ffba9955aede222b9bdd4056183/courgette/disassembler_win32_x64.h
[modify] https://crrev.com/7b221a5ec1175ffba9955aede222b9bdd4056183/courgette/disassembler_win32_x86.cc
[modify] https://crrev.com/7b221a5ec1175ffba9955aede222b9bdd4056183/courgette/disassembler_win32_x86.h

Project Member

Comment 3 by bugdroid1@chromium.org, Jan 13 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8563622217956fce69f0eb03ba7f5676d5441824

commit 8563622217956fce69f0eb03ba7f5676d5441824
Author: huangs <huangs@chromium.org>
Date: Fri Jan 13 18:09:13 2017

[Courgette] Simple AssemblyProgram and Disassembler cleanups.

We perform mundane cleanups to prepare for next step in
AssemblyProgram reduction:
- Delete dead code (static functions) in AssemblyProgram that were left
  over from LabelManager usage.
- Simplify image_base assignment for AssemblyProgram.
  - Source: Disassembler adds virtual function image_base() (PE file
    reads and stores it; ELF just returns 0).
  - Sink: AssemblyProgram takes it on construction (replaces mutator).
- Disassembler::Disassemble() overrides: Rename |target| to |program|.

BUG= 660980 

Review-Url: https://codereview.chromium.org/2583373002
Cr-Commit-Position: refs/heads/master@{#443609}

[modify] https://crrev.com/8563622217956fce69f0eb03ba7f5676d5441824/courgette/adjustment_method_unittest.cc
[modify] https://crrev.com/8563622217956fce69f0eb03ba7f5676d5441824/courgette/assembly_program.cc
[modify] https://crrev.com/8563622217956fce69f0eb03ba7f5676d5441824/courgette/assembly_program.h
[modify] https://crrev.com/8563622217956fce69f0eb03ba7f5676d5441824/courgette/disassembler.h
[modify] https://crrev.com/8563622217956fce69f0eb03ba7f5676d5441824/courgette/disassembler_elf_32.cc
[modify] https://crrev.com/8563622217956fce69f0eb03ba7f5676d5441824/courgette/disassembler_elf_32.h
[modify] https://crrev.com/8563622217956fce69f0eb03ba7f5676d5441824/courgette/disassembler_elf_32_x86_unittest.cc
[modify] https://crrev.com/8563622217956fce69f0eb03ba7f5676d5441824/courgette/disassembler_win32.cc
[modify] https://crrev.com/8563622217956fce69f0eb03ba7f5676d5441824/courgette/disassembler_win32.h
[modify] https://crrev.com/8563622217956fce69f0eb03ba7f5676d5441824/courgette/program_detector.cc

Project Member

Comment 4 by bugdroid1@chromium.org, Mar 1 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/dc779d9f0fab468f70ef7fb2cf5029e2373473dc

commit dc779d9f0fab468f70ef7fb2cf5029e2373473dc
Author: huangs <huangs@chromium.org>
Date: Wed Mar 01 22:22:38 2017

[Courgette] Disassembler code movement and format fix.

We wish to rename ParseAbs32Relocs() and ParseRel32RelocsFromSections(),
and make them part of the Disassembler interface. This requires moving
chunks of code. This CL does the code movement as an independent change
to reduce noise later.

Also including reformatting changes from "git cl format".

BUG= 660980 

Review-Url: https://codereview.chromium.org/2728653003
Cr-Commit-Position: refs/heads/master@{#454068}

[modify] https://crrev.com/dc779d9f0fab468f70ef7fb2cf5029e2373473dc/courgette/disassembler_elf_32.cc
[modify] https://crrev.com/dc779d9f0fab468f70ef7fb2cf5029e2373473dc/courgette/disassembler_elf_32.h
[modify] https://crrev.com/dc779d9f0fab468f70ef7fb2cf5029e2373473dc/courgette/disassembler_win32.cc
[modify] https://crrev.com/dc779d9f0fab468f70ef7fb2cf5029e2373473dc/courgette/disassembler_win32.h

Project Member

Comment 5 by bugdroid1@chromium.org, Mar 22 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c615c911eea856986f8daaab73b7f30860234009

commit c615c911eea856986f8daaab73b7f30860234009
Author: huangs <huangs@chromium.org>
Date: Wed Mar 22 06:52:24 2017

[Courgette] Fix ELF reference sorting.

This CL addresses 2 reference sorting issues in DisassemblerElf32:

(1) Bug fix: In ParseFile(), |abs32_locations_| (RVAs) is translated to
    |abs_offsets| (file offsets), but we sort |abs32_locations_|, which
    is redundant. Actually we should sort |abs_offsets|.
(2) Cleanup: |rel32_relocations_| stores rel32 references sorted by
    RVA, but in ParseFile() we re-sort these in offset order. Previously
    Disassemble() optimizes away redundant sorts, but this makes the
    code less robust. We de-optimize this a little potentially redundant
    sort-by-RVA, to assert that |rel32_locations_| is sorted by RVA
    outside of ParseFile().

This CL also makes Disassemble() more uniform, to prepare for
refactoring in a follow-up. Meanwhile, DisassemblerWin32 does not
experience issue since it assumes RVA order is same as file offset
order (this assumption has not has not caused problems so far).

BUG= 660980 

Review-Url: https://codereview.chromium.org/2744373004
Cr-Commit-Position: refs/heads/master@{#458650}

[modify] https://crrev.com/c615c911eea856986f8daaab73b7f30860234009/courgette/disassembler_elf_32.cc

Project Member

Comment 6 by bugdroid1@chromium.org, Mar 23 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/257f9fb084fd9c3a660ef8d7fd2948cb3d3fd1fb

commit 257f9fb084fd9c3a660ef8d7fd2948cb3d3fd1fb
Author: huangs <huangs@chromium.org>
Date: Thu Mar 23 23:17:50 2017

[Courgette] Refactor: Unify Disassembler::Disassemble() and instantiate AssemblyProgram there.

This is part of AssemblyProgram reduction effort. Details:
- Add Disassembler::Disassemble() and dedup separate PE/ELF versions.
  - Instantiate AssemblyProgram there; replaces old behavior where
    caller (program_detector.cc) instantiates and passes as out param.
- Add virtual Disassembler::GetInstructionGenerator().
  - Remove InstructionGenerator's AssemblyProgram* param; update tests.
- Move InstructionReceptor and InstructionGenerator from AssemblyProgram
  to new file instruction_utils.h.
- Rename ParseAbs32Relocs() to ExtractAbs32Locations(); make common.
- Rename ParseRel32Relocs() to ExtractRel32Locations(); make common.
- Make DisassemblerElf32::abs32_locations_ non-mutable.

BUG= 660980 

Review-Url: https://codereview.chromium.org/2771753004
Cr-Commit-Position: refs/heads/master@{#459272}

[modify] https://crrev.com/257f9fb084fd9c3a660ef8d7fd2948cb3d3fd1fb/courgette/BUILD.gn
[modify] https://crrev.com/257f9fb084fd9c3a660ef8d7fd2948cb3d3fd1fb/courgette/adjustment_method_unittest.cc
[modify] https://crrev.com/257f9fb084fd9c3a660ef8d7fd2948cb3d3fd1fb/courgette/assembly_program.cc
[modify] https://crrev.com/257f9fb084fd9c3a660ef8d7fd2948cb3d3fd1fb/courgette/assembly_program.h
[modify] https://crrev.com/257f9fb084fd9c3a660ef8d7fd2948cb3d3fd1fb/courgette/disassembler.cc
[modify] https://crrev.com/257f9fb084fd9c3a660ef8d7fd2948cb3d3fd1fb/courgette/disassembler.h
[modify] https://crrev.com/257f9fb084fd9c3a660ef8d7fd2948cb3d3fd1fb/courgette/disassembler_elf_32.cc
[modify] https://crrev.com/257f9fb084fd9c3a660ef8d7fd2948cb3d3fd1fb/courgette/disassembler_elf_32.h
[modify] https://crrev.com/257f9fb084fd9c3a660ef8d7fd2948cb3d3fd1fb/courgette/disassembler_elf_32_arm.cc
[modify] https://crrev.com/257f9fb084fd9c3a660ef8d7fd2948cb3d3fd1fb/courgette/disassembler_elf_32_x86.cc
[modify] https://crrev.com/257f9fb084fd9c3a660ef8d7fd2948cb3d3fd1fb/courgette/disassembler_elf_32_x86_unittest.cc
[modify] https://crrev.com/257f9fb084fd9c3a660ef8d7fd2948cb3d3fd1fb/courgette/disassembler_win32.cc
[modify] https://crrev.com/257f9fb084fd9c3a660ef8d7fd2948cb3d3fd1fb/courgette/disassembler_win32.h
[add] https://crrev.com/257f9fb084fd9c3a660ef8d7fd2948cb3d3fd1fb/courgette/instruction_utils.h
[modify] https://crrev.com/257f9fb084fd9c3a660ef8d7fd2948cb3d3fd1fb/courgette/program_detector.cc

Project Member

Comment 7 by bugdroid1@chromium.org, Apr 13 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c4155eb674ece85b214aaf4cb781510d7377d390

commit c4155eb674ece85b214aaf4cb781510d7377d390
Author: huangs <huangs@chromium.org>
Date: Thu Apr 13 20:47:07 2017

[Courgette] Refactor: Store Label Annotation in AssemblyProgram for patch generation.

For Courgette-gen, label adjustment needs lists of abs32 & rel32 Label*
sorted by file offset. Let these lists be "Label Annotations".
Previously these were extracted during label adjustment from list of
instructions in AssemblyProgram, but now we wish to remove these stored
instructions.

This CL make AssemblyProgram store Label annotations. These are computed
only when needed (Courgette-gen / -gen1a). Details:

- Add ParseDetecteExecutableWithAnnotation() alongside
  ParseDetecteExecutable(), to avoid affecting flows that don't require
  Label Annotations.
- AssemblyProgram: Add |*_label_annotations_| as storage vectors. These
  are optionally populated in InstructionStoreReceptor when
  Disassembler::Disassemble() gets called.
  - InstructionCountReceptor now put into use.
- Simplify Label adjustment: AssemblyProgram::HandleInstructionLabels()
  is replaced with visiting AssemblyProgram's Label Annotations.
- Subtle: GraphAdjuster now includes abs64. The class doesn't get used,
  but the change is logical anyway.

BUG= 660980 

Review-Url: https://codereview.chromium.org/2793153003
Cr-Commit-Position: refs/heads/master@{#464536}

[modify] https://crrev.com/c4155eb674ece85b214aaf4cb781510d7377d390/courgette/adjustment_method.cc
[modify] https://crrev.com/c4155eb674ece85b214aaf4cb781510d7377d390/courgette/adjustment_method_2.cc
[modify] https://crrev.com/c4155eb674ece85b214aaf4cb781510d7377d390/courgette/adjustment_method_unittest.cc
[modify] https://crrev.com/c4155eb674ece85b214aaf4cb781510d7377d390/courgette/assembly_program.cc
[modify] https://crrev.com/c4155eb674ece85b214aaf4cb781510d7377d390/courgette/assembly_program.h
[modify] https://crrev.com/c4155eb674ece85b214aaf4cb781510d7377d390/courgette/courgette_tool.cc
[modify] https://crrev.com/c4155eb674ece85b214aaf4cb781510d7377d390/courgette/disassembler.cc
[modify] https://crrev.com/c4155eb674ece85b214aaf4cb781510d7377d390/courgette/disassembler.h
[modify] https://crrev.com/c4155eb674ece85b214aaf4cb781510d7377d390/courgette/disassembler_elf_32_x86_unittest.cc
[modify] https://crrev.com/c4155eb674ece85b214aaf4cb781510d7377d390/courgette/patch_generator_x86_32.h
[modify] https://crrev.com/c4155eb674ece85b214aaf4cb781510d7377d390/courgette/program_detector.cc
[modify] https://crrev.com/c4155eb674ece85b214aaf4cb781510d7377d390/courgette/program_detector.h

Project Member

Comment 8 by bugdroid1@chromium.org, Apr 25 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8b91a4c9f1d554685ee98468c96ee3c374b6633c

commit 8b91a4c9f1d554685ee98468c96ee3c374b6633c
Author: huangs <huangs@chromium.org>
Date: Tue Apr 25 23:04:58 2017

[Courgette] Refactor: Add CourgetteFlow; improve courgette_tool.cc help text.

High level flows in Courgette has messy error-handling code that's
hard to read. This CL adds CourgetteFlow to absorb commonly used data
and error-handling code, and improve readability of high-level logic.
Additional changes:
- Fix Courgette-disadj (debugging flow), which likely was broken by
  http://crrev.com/2793153003; Adjust() is called but AssemblyProgram
  did not annotate Labels.
- Add BasicBuffer interface, implemented by RegionBuffer and (existing)
  BufferedFileReader.
- PatchGeneratorX86_32::Transform(): Implemented TODO to instantiate
  "new" AssemblyProgram to reduce Courgette-gen peak memory.
- courgette_tools.cc: Make help text more uniform.

BUG= 660980 

Review-Url: https://codereview.chromium.org/2827103002
Cr-Commit-Position: refs/heads/master@{#467157}

[modify] https://crrev.com/8b91a4c9f1d554685ee98468c96ee3c374b6633c/courgette/BUILD.gn
[add] https://crrev.com/8b91a4c9f1d554685ee98468c96ee3c374b6633c/courgette/courgette_flow.cc
[add] https://crrev.com/8b91a4c9f1d554685ee98468c96ee3c374b6633c/courgette/courgette_flow.h
[modify] https://crrev.com/8b91a4c9f1d554685ee98468c96ee3c374b6633c/courgette/courgette_tool.cc
[modify] https://crrev.com/8b91a4c9f1d554685ee98468c96ee3c374b6633c/courgette/encode_decode_unittest.cc
[modify] https://crrev.com/8b91a4c9f1d554685ee98468c96ee3c374b6633c/courgette/encoded_program_fuzz_unittest.cc
[modify] https://crrev.com/8b91a4c9f1d554685ee98468c96ee3c374b6633c/courgette/patch_generator_x86_32.h
[modify] https://crrev.com/8b91a4c9f1d554685ee98468c96ee3c374b6633c/courgette/patcher_x86_32.h
[modify] https://crrev.com/8b91a4c9f1d554685ee98468c96ee3c374b6633c/courgette/streams.h

Project Member

Comment 9 by bugdroid1@chromium.org, May 2 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/19281f34d8f96eebe583cd7ccea1da56b9621854

commit 19281f34d8f96eebe583cd7ccea1da56b9621854
Author: huangs <huangs@chromium.org>
Date: Tue May 02 16:59:08 2017

[Courgette] Clean up EnsembleProgram.

Before commiting the main CL for AssemblyProgram reduction, we clean up
EnsenbleProgram to reduce noise. Details:
- Fix includes and comments; run linter and formatter.
- EncodedProgram::EvaluateRel32ARM(): Change non-const ref params to
  pointers.

BUG= 660980 

Review-Url: https://codereview.chromium.org/2858593002
Cr-Commit-Position: refs/heads/master@{#468685}

[modify] https://crrev.com/19281f34d8f96eebe583cd7ccea1da56b9621854/courgette/encoded_program.cc
[modify] https://crrev.com/19281f34d8f96eebe583cd7ccea1da56b9621854/courgette/encoded_program.h

Project Member

Comment 10 by bugdroid1@chromium.org, May 18 2017

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/88451332849bb3632182e4299f4c2b64a240b6c8

commit 88451332849bb3632182e4299f4c2b64a240b6c8
Author: huangs <huangs@chromium.org>
Date: Thu May 18 19:50:34 2017

[Courgette] Reduce AssemblyProgram to reduce Courgette-apply RAM floor and disk churn.

This is the main CL of the AssemblyProgram reduction effort, which makes
Disassembler emit instructions directly to EncodedProgram, and bypass
AssemblyProgram, which becomes a holder of LabelManager and Label
Annotation. Another big change is that Disassembler usage is exposed to
high-level flows (rather than having it hidden inside
ParseDetectedExecutable*()). Details:

- Remove ParseDetectedExecutable*() and expose DetectDisassembler().
- AssemblyProgram: Remove:
  - Entire Instruction class hierarchy.
  - Allocated instruction in the heap (including preallocated DEFBYTE
    instances).
  - enum OP {}, which duplicates EncodedProgram::OP.
  - InstructionVector of pointers, which takes 100's of MB on disk.
  - Encode() interface.
- AssemblyProgram: Add:
  - AnnotateLabels() to generate Label Annotation directly from a
    generator (injected by Disassembler), using LabelReceptor template
    class for counting and storage.
  - PrepareEncodedProgram() to inject data needed (e.g., Label) to
    EncodedProgram.
- Disassembler: Disassemble() is split into CreateProgram() and
  DisassembleAndEncode().
- EncodedProgram: Old AssemblyProgram::Encode() code now moved to
  InstructionStoreReceptor. New GenerateInstruction() is added to
  receive generator from Disassemblyprogram.
- CourgetteFlow: Disassembler explicitly managed:
  - Remove ReadAssemblyProgramFromBuffer().
  - Add ReadDisassemblerFromBuffer().
  - Add CreateEncodedProgramFromDisassemblerAndAssemblyProgram().
- High-level flows in courgette_tool.cc, patch_generator_x86_32.h,
  patcher_x86_32.h: Using new more complex flow that involves
  Disassembler; using ASCII graphics in comment to illustrate.
- Unit tests updates.

Documentation update will be done in a follow-up.

BUG= 660980 

Review-Url: https://codereview.chromium.org/2854113002
Cr-Commit-Position: refs/heads/master@{#472907}

[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/courgette/adjustment_method_unittest.cc
[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/courgette/assembly_program.cc
[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/courgette/assembly_program.h
[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/courgette/courgette_flow.cc
[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/courgette/courgette_flow.h
[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/courgette/courgette_tool.cc
[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/courgette/disassembler.cc
[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/courgette/disassembler.h
[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/courgette/disassembler_elf_32_x86_unittest.cc
[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/courgette/encode_decode_unittest.cc
[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/courgette/encoded_program.cc
[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/courgette/encoded_program.h
[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/courgette/encoded_program_fuzz_unittest.cc
[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/courgette/instruction_utils.h
[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/courgette/patch_generator_x86_32.h
[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/courgette/patcher_x86_32.h
[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/courgette/program_detector.cc
[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/courgette/program_detector.h
[modify] https://crrev.com/88451332849bb3632182e4299f4c2b64a240b6c8/testing/libfuzzer/fuzzers/courgette_fuzzer.cc

Cc: rpop@chromium.org
Status: Fixed (was: Assigned)
Impact of change, from local experiments on Windows 10:

Measurement: Courgette-apply for PGO chrome.7z, 54.0.2840.99 → 55.0.2883.35.

*** RAM Floor ***

Obtained by reducing ProcessMemoryLimit until failure (1 MB granularity). Values stated are minimum for successful Courgette-apply run; for example, "43 MB" means success at 43 MB and failure at 42 MB. Also,
comparing "Old --> New"

32-bit Courgette, 32-bit Chrome:  69 MB --> 43 MB (-38%)
32-bit Courgette, 64-bit Chrome:  60 MB --> 37 MB (-38%)
64-bit Courgette, 32-bit Chrome: 104 MB --> 44 MB (-58%)
64-bit Courgette, 64-bit Chrome:  93 MB --> 36 MB (-61%)

*** TempMapping usage ***

This quantifies Courgette's temp file usage, and corresponds to disk churn under low memory conditions. Experiment uses http://crrev.com/2883773002/ to instrument TempMapping to report total and peak allocation.

Total:
32-bit Courgette, 32-bit Chrome: 2010.7 MB --> 1239.3 MB (-38%)
32-bit Courgette, 64-bit Chrome: 2489.7 MB --> 1483.1 MB (-40%)
64-bit Courgette, 32-bit Chrome: 2782.4 MB --> 1238.9 MB (-55%)
64-bit Courgette, 64-bit Chrome: 3496.3 MB --> 1483.1 MB (-58%)

Peak:
32-bit Courgette, 32-bit Chrome: 469.8 MB --> 430.6 MB (-8%)
32-bit Courgette, 64-bit Chrome: 494.0 MB --> 494.0 MB (0%)
64-bit Courgette, 32-bit Chrome: 872.4 MB --> 430.6 MB (-50%)
64-bit Courgette, 64-bit Chrome: 880.8 MB --> 494.0 MB (-44%)

*** Summary ***

- The change reduces RAM floor by ~40% for 32-bit Courgette, and ~60% for 64-bit Courgette.
- The change reduces total disk churn by ~40% for 32-bit Courgette, and ~55% for 64-bit Courgette.
- The change does not affect peak disk use for 32-bit Courgette, likely because peak occurs elsewhere (final patching step).
- The change reduces peak disk use for 64-bit Courgette by ~45% for 64-bit Courgette.

Sign in to add a comment