Track pak file size in supersize |
||||||||||
Issue descriptionIdeally .size files would contain entries that cover every byte that makes up Chrome.apk. This bug covers tracking the bytes that go into .pak files. Question 1: Do we want to bother with tracking compressed size for .pak files (since locale paks are stored compressed). I think the answer here is "no", mainly because it's hard. resource_size.py tracks compressed size on a higher level (not on a per-symbol level), so there is already some coverage for this. Question 2: What exactly should we store for .pak entries? As a first guess: name = "IDR_FOO" full_name = "path/to/foo.grdp: IDR_FOO" size = size_of_entry padding = 6 (table-of-contents entry size = 2 byte ID + 4 byte offset) section_name = "pak.nontranslated" / "pak.translations" object_path = path/to/file.o source_path = path/to/file.cc section_name: based on whether file came from a locale pak vs normal pak object_path: can be extracted from .o.whitelist files. full_name: * Arguably the grd path should be stored as a separate field, but it will save us updating the file format to stick it in here :). * We could map from numerical ID -> .grd file using the mapping defined here: https://cs.chromium.org/chromium/src/tools/gritsettings/resource_ids * I think it would be even better to map to .grdp file though, and I don't think there's currently a way to do that. Ideas: 1) Have grit output an extra file with this information when it's building .pak files. 2) Have supersize run a new grit command that given a .grd file outputs a mapping of .grdp -> ID If an IDR_ entry is referenced by multiple .o.whitelist files, then we should apply the same Ancestor logic to it. Note: We don't store the numeric ID value anywhere, and I think that's fine (I see no use for it).
,
May 18 2017
Actually... we can just make our padding computation dependent on which section the symbol is in. E.g. in CalculatePadding(), we could blinding give all pak symbols a padding of 6.
,
Jun 16 2017
Note: There now exists a tool for listing .pak contents at //tools/grit/pak_util.py
,
Aug 10 2017
,
Sep 5 2017
,
Sep 11 2017
,
Sep 11 2017
Sample output from https://chromium-review.googlesource.com/c/chromium/src/+/650652 attached. (posted on wrong bug before)
,
Sep 18 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/266d0b590296cb874ef0ea38745cc6acc37bb92c commit 266d0b590296cb874ef0ea38745cc6acc37bb92c Author: Peter Wen <wnwen@chromium.org> Date: Mon Sep 18 14:24:48 2017 Grit: Add pak.info mapping id to source grd file Knowing which ids map to which names and grd or grdp file is important for attributing pak file sizes. This information will then be processed to be visible through the supersize tool. BUG= 723672 Change-Id: I218faf7f786174f93d5518e90917ed0656e6e4ce Reviewed-on: https://chromium-review.googlesource.com/650652 Commit-Queue: Peter Wen <wnwen@chromium.org> Reviewed-by: Lei Zhang <thestig@chromium.org> Reviewed-by: Andrew Grieve <agrieve@chromium.org> Cr-Commit-Position: refs/heads/master@{#502573} [modify] https://crrev.com/266d0b590296cb874ef0ea38745cc6acc37bb92c/tools/grit/grit/format/data_pack.py [modify] https://crrev.com/266d0b590296cb874ef0ea38745cc6acc37bb92c/tools/grit/grit/grd_reader.py [modify] https://crrev.com/266d0b590296cb874ef0ea38745cc6acc37bb92c/tools/grit/grit/grd_reader_unittest.py [modify] https://crrev.com/266d0b590296cb874ef0ea38745cc6acc37bb92c/tools/grit/grit/node/base.py [modify] https://crrev.com/266d0b590296cb874ef0ea38745cc6acc37bb92c/tools/grit/grit/tool/build.py [modify] https://crrev.com/266d0b590296cb874ef0ea38745cc6acc37bb92c/tools/grit/grit_rule.gni [modify] https://crrev.com/266d0b590296cb874ef0ea38745cc6acc37bb92c/tools/grit/repack.gni
,
Sep 22 2017
hey, i might be completely wrong but I'm debugging a performance regression on chrome os. It turns out there are *.pak.info files in addition to *.pak in the directory /opt/google/chrome/locales and each pak.info file size is on average 3 times of its corresponding pak file. any thoughts why?
,
Sep 22 2017
These .pak.info files are meant only for debugging purposes, so they certainly should not be included on chrome os images. I have no idea how chromeos packaging works though.
,
Sep 29 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0e7a8e94044efb3e15b86eb379d77648c8150c39 commit 0e7a8e94044efb3e15b86eb379d77648c8150c39 Author: Peter Wen <wnwen@chromium.org> Date: Fri Sep 29 00:18:58 2017 Grit: Fix pak.info line termination When pak.info files are concatenated whenever pak files are merged, they need to have a consistent format so that the concatenated file does not have merged lines. BUG= 723672 Change-Id: Ic45f62d0320fe9b167793694fc13feec76e336ec Reviewed-on: https://chromium-review.googlesource.com/690317 Commit-Queue: Peter Wen <wnwen@chromium.org> Reviewed-by: Lei Zhang <thestig@chromium.org> Cr-Commit-Position: refs/heads/master@{#505225} [modify] https://crrev.com/0e7a8e94044efb3e15b86eb379d77648c8150c39/tools/grit/grit/tool/build.py
,
Nov 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e3942dce2d630e4ce1dfadaca403d69b7627e2c2 commit e3942dce2d630e4ce1dfadaca403d69b7627e2c2 Author: Peter Wen <wnwen@chromium.org> Date: Wed Nov 08 15:39:22 2017 Android: Create *.apk.pak.info files These *.apk.pak.info files will be used by supersize. Bug: 723672 Change-Id: Ia275d755d3059ec2c957b4fd09da84526dbc2363 Reviewed-on: https://chromium-review.googlesource.com/749505 Commit-Queue: Peter Wen <wnwen@chromium.org> Reviewed-by: agrieve <agrieve@chromium.org> Cr-Commit-Position: refs/heads/master@{#514838} [modify] https://crrev.com/e3942dce2d630e4ce1dfadaca403d69b7627e2c2/build/android/gyp/apkbuilder.py [modify] https://crrev.com/e3942dce2d630e4ce1dfadaca403d69b7627e2c2/build/config/android/internal_rules.gni [modify] https://crrev.com/e3942dce2d630e4ce1dfadaca403d69b7627e2c2/build/config/android/rules.gni
,
Nov 16 2017
,
Nov 23 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/7c8647953491525c34f91ed54198b1c1a9ef0ec8 commit 7c8647953491525c34f91ed54198b1c1a9ef0ec8 Author: Peter Wen <wnwen@chromium.org> Date: Thu Nov 23 20:02:32 2017 Supersize: Add pak sizes Add uncompressed sizes for all pak files, locale pak entries are all combined and summed to the same identifier. Currently source_path is the grd/grdp file and the object_path is the actual pak file path in the apk. Accounts for almost all the bytes in pak files (ignoring the global overhead in each pak file of 18 bytes). Bug: 723672 Change-Id: Ic7266fa8763b4a382ceb54bea836a18453619158 Reviewed-on: https://chromium-review.googlesource.com/779879 Commit-Queue: Peter Wen <wnwen@chromium.org> Reviewed-by: agrieve <agrieve@chromium.org> Cr-Commit-Position: refs/heads/master@{#519010} [modify] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/archive.py [modify] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/console.py [modify] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/describe.py [modify] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/file_format.py [modify] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/integration_test.py [modify] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/linker_map_parser.py [modify] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/models.py [modify] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/nm.py [modify] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/testdata/Archive.golden [modify] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/testdata/Archive_Elf.golden [modify] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/testdata/Archive_OutputDirectory.golden [add] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/testdata/Archive_Pak.golden [modify] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/testdata/Console.golden [modify] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/testdata/Csv.golden [modify] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/testdata/Diff_Basic.golden [modify] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/testdata/FullDescription.golden [modify] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/testdata/SymbolGroupMethods.golden [add] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/testdata/mock_output_directory/en-US.pak [add] https://crrev.com/7c8647953491525c34f91ed54198b1c1a9ef0ec8/tools/binary_size/libsupersize/testdata/mock_output_directory/en-US.pak.info
,
Nov 29 2017
Still todo: map .o.whitelist entries to resource ids
,
Dec 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/dbcf1b4e167e960c0776b1d845070247e9ff5fd4 commit dbcf1b4e167e960c0776b1d845070247e9ff5fd4 Author: Peter Wen <wnwen@chromium.org> Date: Wed Dec 06 17:18:02 2017 Supersize: Corelate pak symbols to source files Use object_path and source_path from .o.whitelist files to find the right paths for each pak symbol. Refactor and generalize aliases/coalescing for all symbols. Bug: 723672 Change-Id: I8a0385241ffc60334bee981a98566e46954884d7 Reviewed-on: https://chromium-review.googlesource.com/807147 Commit-Queue: Peter Wen <wnwen@chromium.org> Reviewed-by: agrieve <agrieve@chromium.org> Cr-Commit-Position: refs/heads/master@{#522120} [modify] https://crrev.com/dbcf1b4e167e960c0776b1d845070247e9ff5fd4/tools/binary_size/libsupersize/archive.py [modify] https://crrev.com/dbcf1b4e167e960c0776b1d845070247e9ff5fd4/tools/binary_size/libsupersize/integration_test.py [modify] https://crrev.com/dbcf1b4e167e960c0776b1d845070247e9ff5fd4/tools/binary_size/libsupersize/ninja_parser.py [modify] https://crrev.com/dbcf1b4e167e960c0776b1d845070247e9ff5fd4/tools/binary_size/libsupersize/testdata/Archive_Pak.golden
,
Dec 6 2017
I think this bug is done?
,
Jan 4 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/2bbfe472e8c34e6023f1eb404377d7e7371e2154 commit 2bbfe472e8c34e6023f1eb404377d7e7371e2154 Author: Andrew Grieve <agrieve@chromium.org> Date: Thu Jan 04 17:02:18 2018 Supersize: Add pak and other support to html_report Other html_report tweaks: * Changes --include-symbols to --symbol-size-threshold * This is useful because it's nice to see the actual symbol names for pak files (and other large symbols) * Hides .other symbols by default (since they are exist at /) * But they can be shown via "options" in the UI * Adds a ?sections= query parameter so you can link to a chart with just certain sections. Also: * Adds SymbolGroup.WhereIsPak() and WhereIsNative() * Adds canned_query.PakByPath() * Adds models.BaseSizeInfo * Which has: .symbols, .native_symbols, .pak_symbols Bug: 723672 , 793014 Change-Id: Ia054826a1d52240deb979834b45db940e956b716 Reviewed-on: https://chromium-review.googlesource.com/849315 Commit-Queue: agrieve <agrieve@chromium.org> Reviewed-by: Peter Wen <wnwen@chromium.org> Cr-Commit-Position: refs/heads/master@{#527016} [modify] https://crrev.com/2bbfe472e8c34e6023f1eb404377d7e7371e2154/tools/binary_size/libsupersize/canned_queries.py [modify] https://crrev.com/2bbfe472e8c34e6023f1eb404377d7e7371e2154/tools/binary_size/libsupersize/describe.py [modify] https://crrev.com/2bbfe472e8c34e6023f1eb404377d7e7371e2154/tools/binary_size/libsupersize/diff.py [modify] https://crrev.com/2bbfe472e8c34e6023f1eb404377d7e7371e2154/tools/binary_size/libsupersize/html_report.py [modify] https://crrev.com/2bbfe472e8c34e6023f1eb404377d7e7371e2154/tools/binary_size/libsupersize/integration_test.py [modify] https://crrev.com/2bbfe472e8c34e6023f1eb404377d7e7371e2154/tools/binary_size/libsupersize/models.py [modify] https://crrev.com/2bbfe472e8c34e6023f1eb404377d7e7371e2154/tools/binary_size/libsupersize/template/D3SymbolTreeMap.js [modify] https://crrev.com/2bbfe472e8c34e6023f1eb404377d7e7371e2154/tools/binary_size/libsupersize/template/index.html [modify] https://crrev.com/2bbfe472e8c34e6023f1eb404377d7e7371e2154/tools/binary_size/libsupersize/testdata/Console.golden |
||||||||||
►
Sign in to add a comment |
||||||||||
Comment 1 by agrieve@chromium.org
, May 17 2017