New issue
Advanced search Search tips

Issue 648864 link

Starred by 2 users

Issue metadata

Status: Verified
Owner:
Closed: Sep 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocked on:
issue v8:5412

Blocking:
issue 648610



Sign in to add a comment

All V8 executables and tests should properly support component=shared_library mode

Project Member Reported by machenb...@chromium.org, Sep 21 2016

Issue description

A V8 debug build archive (d8, cctest, unittests and all fuzzers) is tremendously large with static linking. They should all support sharing the same libraries to save space.

Example of arm debug executables:
https://isolateserver.appspot.com/browse?namespace=default-gzip&hash=3261e5b13bc6e7c3f0f938c37c6685d730a7bebc

This is maybe blocking issue 648610.
 
Labels: -Pri-2 Pri-1
This also impacts Chromium. All our executables are in all archives. E.g.
https://build.chromium.org/p/chromium.linux/builders/Linux%20Builder%20%28dbg%29/builds/104319/steps/package%20build/logs/stdio

The archive of one debug build is 7GB. I think we can chop off ~2GB from this by fixing this issue. The overall savings for upload/download/storage across the whole fleet would be ... quite large.

It also currently doesn't scale because the number of fuzzers is growing.
I also think cctest and unittests can be disregarded. The fuzzers are the main problem.

Comment 3 by jochen@chromium.org, Sep 21 2016

d8 is fixed now, no?

I'd expect that switching the fuzzers and unittests will be easy...
Cc: -jgruber@chromium.org
Owner: jgruber@chromium.org
Status: Assigned (was: Untriaged)
I can take a look at this.
Blockedon: v8:5412
For the record, also mksnapshot might be able to share the same library with v8_shell in the snapshot toolchain or with all the executables if snapshot toolchain == target toolchain.

Also v8_parser_shell is large.
Blocking: 648610
Project Member

Comment 8 by bugdroid1@chromium.org, Sep 26 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/22606f0c29421d6a72e46595ec2995c7771ff070

commit 22606f0c29421d6a72e46595ec2995c7771ff070
Author: jgruber <jgruber@chromium.org>
Date: Mon Sep 26 07:40:24 2016

Enable component builds for fuzzers

V8 is collecting a growing amount of fuzzers, all of which take substantial
space on the bots and in chromium build archives. This CL improves that
situation by allowing component (shared library) builds for almost all fuzzers.

The parser fuzzer is handled as an exception since it would require exporting a
large number of additional functions.

A component build results in about a 50-100x improvement in file size for each
fuzzer (~50M-100M to around 1.1M).

BUG= chromium:648864 
CQ_INCLUDE_TRYBOTS=master.tryserver.chromium.linux:linux_chromium_compile_dbg_ng;master.tryserver.chromium.android:android_clang_dbg_recipe

Review-Url: https://codereview.chromium.org/2360983002
Cr-Commit-Position: refs/heads/master@{#39709}

[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/BUILD.gn
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/src/assert-scope.h
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/src/contexts.h
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/src/execution.h
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/src/factory.h
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/src/handles.h
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/src/lookup.h
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/src/machine-type.h
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/src/objects.h
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/src/regexp/jsregexp.h
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/src/wasm/ast-decoder.h
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/src/wasm/encoder.h
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/src/wasm/module-decoder.h
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/src/wasm/wasm-interpreter.h
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/src/wasm/wasm-js.h
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/src/wasm/wasm-module.h
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/src/wasm/wasm-result.h
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/src/zone/accounting-allocator.h
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/src/zone/zone.h
[modify] https://crrev.com/22606f0c29421d6a72e46595ec2995c7771ff070/test/fuzzer/fuzzer.gyp

Status: Fixed (was: Assigned)
After the fuzzer commit (see above), there are only a few remaining large files:

cctest
unittests
libv8.so
mksnapshot
generate-bytecode-expectations
v8_parser_shell
v8_simple_parser_fuzzer

Of these, 

* we can't do anything about libv8.so; 
* cctest, v8_parser_shell and v8_simple_parser_fuzzer use too many internal functions to make component builds practical;

That leaves unittests, mksnapshot and generate-bytecode-expectations for potential savings.

Closing this issue since the bots should now be able to handle our archive with fuzzer file size reductions. Opened issue 650171 for the remaining binaries.
Project Member

Comment 10 by bugdroid1@chromium.org, Sep 26 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/v8/v8.git/+/d85a349771482333e5f9b5a0febf57f77a48ab00

commit d85a349771482333e5f9b5a0febf57f77a48ab00
Author: machenbach <machenbach@chromium.org>
Date: Mon Sep 26 10:22:11 2016

[build] Enable shared libraries for arm debug

BUG= chromium:648864 ,chromium:648610
NOTRY=true
TBR=jgruber@chromium.org

Review-Url: https://codereview.chromium.org/2371713002
Cr-Commit-Position: refs/heads/master@{#39712}

[modify] https://crrev.com/d85a349771482333e5f9b5a0febf57f77a48ab00/infra/mb/mb_config.pyl

Status: Verified (was: Fixed)
Thanks a lot for fixing this!

I left some comments on the CL about the impact of this across the fleet:
https://codereview.chromium.org/2360983002

Sign in to add a comment