Implement XZ compression for delta payloads. |
||||
Issue descriptionCurrently the client support XZ decompression operation, but we are not creating any XZ operations for delta payloads. XZ can reduce the delta size potentially by like 10%-20%. We just need to implement the XZ compression for the CrOS side. AOSP already does this. This is only for Delta payloads since we are already stuck with full payload major version 1 for at least another three years.
,
Dec 13 2017
,
Dec 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/939ebbf4579eb1cf6110bd757a646b901c42f21b commit 939ebbf4579eb1cf6110bd757a646b901c42f21b Author: Amin Hassani <ahassani@google.com> Date: Thu Dec 14 01:26:04 2017 update_payload: Add dependency to backports-lzma backports-lzma is needed for python 2.7 for checking REPLACE_XZ operations in payloads. lzma is included in python 3.3 and onwards. BUG= chromium:758792 TEST=unittests pass; paycheck.py with a xz generated payload pass; CQ-DEPEND=CL:823217 Change-Id: I1f41902061d7ed15791e6b48f6786fa8c77a9d50 Reviewed-on: https://chromium-review.googlesource.com/823234 Commit-Ready: Amin Hassani <ahassani@chromium.org> Tested-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Chirantan Ekbote <chirantan@chromium.org> [modify] https://crrev.com/939ebbf4579eb1cf6110bd757a646b901c42f21b/chromeos-base/update_payload/update_payload-9999.ebuild
,
Dec 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/portage-stable/+/22fff8a34130a37eab116170982dae09a10252fc commit 22fff8a34130a37eab116170982dae09a10252fc Author: Amin Hassani <ahassani@google.com> Date: Thu Dec 14 01:26:04 2017 backports-lzma: upgraded package to upstream import dev-python/backports-lzma version 0.0.8 BUG= chromium:758792 TEST=unittests pass; Change-Id: I2d5a25fae6a850d17606bfc7dc3c4470e2bb14bd Reviewed-on: https://chromium-review.googlesource.com/823217 Commit-Ready: Amin Hassani <ahassani@chromium.org> Tested-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Chirantan Ekbote <chirantan@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [add] https://crrev.com/22fff8a34130a37eab116170982dae09a10252fc/metadata/md5-cache/dev-python/backports-lzma-0.0.8 [add] https://crrev.com/22fff8a34130a37eab116170982dae09a10252fc/dev-python/backports-lzma/backports-lzma-0.0.8.ebuild [add] https://crrev.com/22fff8a34130a37eab116170982dae09a10252fc/dev-python/backports-lzma/Manifest [add] https://crrev.com/22fff8a34130a37eab116170982dae09a10252fc/dev-python/backports-lzma/metadata.xml
,
Dec 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/aosp/platform/system/update_engine/+/717c324c334cc3bac954101f7a127fe8ef56e4e6 commit 717c324c334cc3bac954101f7a127fe8ef56e4e6 Author: Amin Hassani <ahassani@google.com> Date: Thu Dec 14 23:26:58 2017 update_engine: Add XZ compression for CrOS This patch implements XZ compression for Chrome OS. It is only activated for only delta payload at this point. The XZ compression is typically slower than Bzip2 to compress, but faster to decompress. However, it uses slightly higher memory in decompression. It compresses about 5%-10% better than the current Bzip2. BUG= chromium:758792 TEST=unittests pass; brillo_update_paylod {generate|verify} passes; Change-Id: Icdaf1d37d344c30a796845eb64daa00120f3598a Reviewed-on: https://chromium-review.googlesource.com/811895 Commit-Ready: Amin Hassani <ahassani@chromium.org> Tested-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Alex Deymo <deymo@chromium.org> Reviewed-by: Ben Chan <benchan@chromium.org> [modify] https://crrev.com/717c324c334cc3bac954101f7a127fe8ef56e4e6/payload_generator/zip_unittest.cc [modify] https://crrev.com/717c324c334cc3bac954101f7a127fe8ef56e4e6/update_engine.gyp [modify] https://crrev.com/717c324c334cc3bac954101f7a127fe8ef56e4e6/payload_generator/xz_chromeos.cc
,
Dec 14 2017
The following revision refers to this bug: https://chromium.googlesource.com/aosp/platform/system/update_engine/+/f1d6ceaedbcf81044d6af6b716c63ed79996f0a4 commit f1d6ceaedbcf81044d6af6b716c63ed79996f0a4 Author: Amin Hassani <ahassani@google.com> Date: Thu Dec 14 23:26:58 2017 update_payload: Add XZ compression support This patch adds support for checking a payload that has REPLACE_XZ operations. REPLACE_XZ was added in minor version 3. BUG= chromium:758792 TEST=unittests pass; paycheck.py with a xz generated payload pass; CQ-DEPEND=CL:823234 Change-Id: If82c767a201b189e464f459d5c19485e5278d9b1 Reviewed-on: https://chromium-review.googlesource.com/823227 Commit-Ready: Amin Hassani <ahassani@chromium.org> Tested-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Ben Chan <benchan@chromium.org> Reviewed-by: Sen Jiang <senj@chromium.org> [modify] https://crrev.com/f1d6ceaedbcf81044d6af6b716c63ed79996f0a4/scripts/update_payload/checker_unittest.py [modify] https://crrev.com/f1d6ceaedbcf81044d6af6b716c63ed79996f0a4/scripts/update_payload/test_utils.py [modify] https://crrev.com/f1d6ceaedbcf81044d6af6b716c63ed79996f0a4/scripts/update_payload/applier.py [modify] https://crrev.com/f1d6ceaedbcf81044d6af6b716c63ed79996f0a4/scripts/update_payload/checker.py
,
Dec 15 2017
The change in #6 passed the CQ and has been merged but is now causing all canary builds to fail with "ImportError: No module named backports" in Paygen stage. One example: https://luci-milo.appspot.com/buildbot/chromeos/arkham-release/1762
,
Dec 15 2017
Sorry for the trouble. I'm going to see if I can fix this issue. If not I'll just revert the last two CLs. That should fix it for now.
,
Dec 15 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/bdaf4fb857064a77be4c28585c480375e75d1d14 commit bdaf4fb857064a77be4c28585c480375e75d1d14 Author: Amin Hassani <ahassani@google.com> Date: Fri Dec 15 23:52:49 2017 chromium-os-sdk: Add backports-lzma as a dependency chromite needs backports-lzma to run paygenbuild and paygentest autotests on hosts. BUG= chromium:758792 TEST=tryjob Change-Id: Iae8137a0b21b12088f5cfde59b9edf1487bd1d50 Reviewed-on: https://chromium-review.googlesource.com/830254 Commit-Ready: Amin Hassani <ahassani@chromium.org> Tested-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> [rename] https://crrev.com/bdaf4fb857064a77be4c28585c480375e75d1d14/virtual/target-chromium-os-sdk/target-chromium-os-sdk-1-r93.ebuild [modify] https://crrev.com/bdaf4fb857064a77be4c28585c480375e75d1d14/virtual/target-chromium-os-sdk/target-chromium-os-sdk-1.ebuild
,
Dec 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/aosp/platform/system/update_engine/+/b44f73baa8d302a077ab1c0c73eac63bd02350e5 commit b44f73baa8d302a077ab1c0c73eac63bd02350e5 Author: Amin Hassani <ahassani@chromium.org> Date: Sat Dec 16 06:19:46 2017 Revert "update_payload: Add XZ compression support" This reverts commit f1d6ceaedbcf81044d6af6b716c63ed79996f0a4. Reason for revert: <canaries failing> Original change's description: > update_payload: Add XZ compression support > > This patch adds support for checking a payload that has REPLACE_XZ > operations. REPLACE_XZ was added in minor version 3. > > BUG= chromium:758792 > TEST=unittests pass; paycheck.py with a xz generated payload pass; > CQ-DEPEND=CL:823234 > > Change-Id: If82c767a201b189e464f459d5c19485e5278d9b1 > Reviewed-on: https://chromium-review.googlesource.com/823227 > Commit-Ready: Amin Hassani <ahassani@chromium.org> > Tested-by: Amin Hassani <ahassani@chromium.org> > Reviewed-by: Ben Chan <benchan@chromium.org> > Reviewed-by: Sen Jiang <senj@chromium.org> TBR=benchan@chromium.org,senj@chromium.org,ahassani@chromium.org Change-Id: I931ff3be081a41fe5fceef0e049ba4165c6acb49 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: chromium:795313 Reviewed-on: https://chromium-review.googlesource.com/830054 Commit-Ready: Amin Hassani <ahassani@chromium.org> Tested-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Ben Chan <benchan@chromium.org> [modify] https://crrev.com/b44f73baa8d302a077ab1c0c73eac63bd02350e5/scripts/update_payload/checker_unittest.py [modify] https://crrev.com/b44f73baa8d302a077ab1c0c73eac63bd02350e5/scripts/update_payload/test_utils.py [modify] https://crrev.com/b44f73baa8d302a077ab1c0c73eac63bd02350e5/scripts/update_payload/applier.py [modify] https://crrev.com/b44f73baa8d302a077ab1c0c73eac63bd02350e5/scripts/update_payload/checker.py
,
Dec 16 2017
The following revision refers to this bug: https://chromium.googlesource.com/aosp/platform/system/update_engine/+/c77e3a2d157aba4a016da5879809312c74c9f6bf commit c77e3a2d157aba4a016da5879809312c74c9f6bf Author: Amin Hassani <ahassani@chromium.org> Date: Sat Dec 16 21:09:31 2017 Revert "update_engine: Add XZ compression for CrOS" This reverts commit 717c324c334cc3bac954101f7a127fe8ef56e4e6. Reason for revert: <canaries failed> Original change's description: > update_engine: Add XZ compression for CrOS > > This patch implements XZ compression for Chrome OS. It is only activated > for only delta payload at this point. The XZ compression is typically > slower than Bzip2 to compress, but faster to decompress. However, it > uses slightly higher memory in decompression. It compresses about 5%-10% > better than the current Bzip2. > > BUG= chromium:758792 > TEST=unittests pass; brillo_update_paylod {generate|verify} passes; > > Change-Id: Icdaf1d37d344c30a796845eb64daa00120f3598a > Reviewed-on: https://chromium-review.googlesource.com/811895 > Commit-Ready: Amin Hassani <ahassani@chromium.org> > Tested-by: Amin Hassani <ahassani@chromium.org> > Reviewed-by: Alex Deymo <deymo@chromium.org> > Reviewed-by: Ben Chan <benchan@chromium.org> TBR=benchan@chromium.org,deymo@chromium.org,deymo@google.com,senj@chromium.org,ahassani@chromium.org Change-Id: I43e8c5ebec84d395b87aa782fb4d478a5fe99279 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: chromium:795313 Reviewed-on: https://chromium-review.googlesource.com/830053 Commit-Ready: Amin Hassani <ahassani@chromium.org> Tested-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Ben Chan <benchan@chromium.org> [modify] https://crrev.com/c77e3a2d157aba4a016da5879809312c74c9f6bf/payload_generator/zip_unittest.cc [modify] https://crrev.com/c77e3a2d157aba4a016da5879809312c74c9f6bf/update_engine.gyp [modify] https://crrev.com/c77e3a2d157aba4a016da5879809312c74c9f6bf/payload_generator/xz_chromeos.cc
,
Jan 23 2018
The following revision refers to this bug: https://chromium.googlesource.com/aosp/platform/system/update_engine/+/0de7f7847ff2a101a905b75d2ed0867d5ac8fba7 commit 0de7f7847ff2a101a905b75d2ed0867d5ac8fba7 Author: Amin Hassani <ahassani@google.com> Date: Tue Jan 23 09:48:49 2018 Relanding 'update_payload: Add XZ compression support' This patch adds support for checking a payload that has REPLACE_XZ operations. REPLACE_XZ was added in minor version 3. BUG= chromium:758792 TEST=unittests pass; paycheck.py with a xz generated payload pass; CQ-DEPEND=CL:823234 Change-Id: I6ec8068e233f2d595fda93a985923d85c59f150e Reviewed-on: https://chromium-review.googlesource.com/872124 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Ben Chan <benchan@chromium.org> Reviewed-by: Sen Jiang <senj@chromium.org> [modify] https://crrev.com/0de7f7847ff2a101a905b75d2ed0867d5ac8fba7/scripts/update_payload/checker_unittest.py [modify] https://crrev.com/0de7f7847ff2a101a905b75d2ed0867d5ac8fba7/scripts/update_payload/test_utils.py [modify] https://crrev.com/0de7f7847ff2a101a905b75d2ed0867d5ac8fba7/scripts/update_payload/applier.py [modify] https://crrev.com/0de7f7847ff2a101a905b75d2ed0867d5ac8fba7/scripts/update_payload/checker.py
,
Jan 26 2018
The following revision refers to this bug: https://chromium.googlesource.com/aosp/platform/system/update_engine/+/43900aea4dc349faa12d60ba74d363640023719f commit 43900aea4dc349faa12d60ba74d363640023719f Author: Amin Hassani <ahassani@google.com> Date: Fri Jan 26 03:24:00 2018 Relanding 'update_engine: Add XZ compression for CrOS' This patch implements XZ compression for Chrome OS. It is only activated for only delta payload at this point. The XZ compression is typically slower than Bzip2 to compress, but faster to decompress. However, it uses slightly higher memory in decompression. It compresses about 5%-10% better than the current Bzip2. BUG= chromium:758792 TEST=unittests pass; brillo_update_paylod {generate|verify} passes; Change-Id: I2fd8f6af30449c779cd2abb061e42ab389913d93 Reviewed-on: https://chromium-review.googlesource.com/872125 Commit-Ready: Amin Hassani <ahassani@chromium.org> Tested-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Sen Jiang <senj@chromium.org> [modify] https://crrev.com/43900aea4dc349faa12d60ba74d363640023719f/payload_generator/zip_unittest.cc [modify] https://crrev.com/43900aea4dc349faa12d60ba74d363640023719f/update_engine.gyp [modify] https://crrev.com/43900aea4dc349faa12d60ba74d363640023719f/payload_generator/xz_chromeos.cc
,
May 10 2018
|
||||
►
Sign in to add a comment |
||||
Comment 1 by ahass...@chromium.org
, Dec 6 2017