New issue
Advanced search Search tips
Starred by 1 user
Status: Fixed
Owner: ----
Closed: Oct 20
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment
PaygenBuild* steps failing due to missing bspatch binary
Project Member Reported by kitching@google.com, Sep 1 Back to list
Here's a sample stacktrace from the last lulu failure:


https://uberchromegw.corp.google.com/i/chromeos/builders/lulu-release/builds/1455/steps/PaygenBuildCanary/logs/stdio


22:49:46: ERROR: <type 'exceptions.OSError'>: [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/lib/parallel.py", line 602, in TaskRunner
    task(*x, **task_kwargs)
  File "/b/c/cbuild/repository/chromite/lib/parallel.py", line 800, in <lambda>
    fn = lambda idx, task_args: out_queue.put((idx, task(*task_args)))
  File "/b/c/cbuild/repository/chromite/lib/paygen/paygen_build_lib.py", line 251, in _GenerateSinglePayload
    dry_run=dry_run)
  File "/b/c/cbuild/repository/chromite/lib/paygen/paygen_payload_lib.py", line 827, in CreateAndUploadPayload
    dry_run=dry_run).Run()
  File "/b/c/cbuild/repository/chromite/lib/paygen/paygen_payload_lib.py", line 707, in Run
    self._drm(self._VerifyPayload)
  File "/b/c/cbuild/repository/chromite/lib/paygen/dryrun_lib.py", line 45, in __call__
    return self.Run(func, *args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/paygen/dryrun_lib.py", line 82, in Run
    return self._Call(func, *args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/paygen/dryrun_lib.py", line 86, in _Call
    return func(*args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/paygen/paygen_payload_lib.py", line 681, in _VerifyPayload
    self._ApplyPayload(payload, is_delta)
  File "/b/c/cbuild/repository/chromite/lib/paygen/paygen_payload_lib.py", line 641, in _ApplyPayload
    payload.Apply(bspatch_path=bspatch_path, **part_files)
  File "/b/c/cbuild/repository/src/platform/dev/host/lib/update_payload/payload.py", line 321, in Apply
    old_rootfs_part=old_rootfs_part)
  File "/b/c/cbuild/repository/src/platform/dev/host/lib/update_payload/applier.py", line 569, in Run
    self.payload.manifest.old_rootfs_info)
  File "/b/c/cbuild/repository/src/platform/dev/host/lib/update_payload/applier.py", line 517, in _ApplyToPartition
    new_part_file, new_part_info.size)
  File "/b/c/cbuild/repository/src/platform/dev/host/lib/update_payload/applier.py", line 468, in _ApplyOperations
    new_part_file)
  File "/b/c/cbuild/repository/src/platform/dev/host/lib/update_payload/applier.py", line 389, in _ApplySourceBsdiffOperation
    subprocess.check_call(bspatch_cmd)
  File "/usr/lib/python2.7/subprocess.py", line 535, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 522, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory


applier.py:387-389:
      bspatch_cmd = [self.bspatch_path, old_file_name, new_file_name,
                     patch_file_name, in_extents_arg, out_extents_arg]
      subprocess.check_call(bspatch_cmd)


Going a little up the stack, there is a "paygen_build_lib.py" being kicked off by some task queue.  This file seems to call into paygen_payload_lib.CreateAndUploadPayload, whose first step is to download and unzip this "au-generator.zip", putting its binaries in PATH.


Then we jump from paygen_*_lib into upload_payload/applier.py, which uses bspatch (which looks for it in generator_dir, which is where au-generator.zip was extracted):
  File "/b/c/cbuild/repository/chromite/lib/paygen/paygen_payload_lib.py", line 641, in _ApplyPayload
    bspatch_path = os.path.join(self.generator_dir, 'bspatch')
    try:
      payload.Apply(bspatch_path=bspatch_path, **part_files)
  File "/b/c/cbuild/repository/src/platform/dev/host/lib/update_payload/payload.py", line 321, in Apply
    old_rootfs_part=old_rootfs_part)
  File "/b/c/cbuild/repository/src/platform/dev/host/lib/update_payload/applier.py", line 569, in Run
    self.payload.manifest.old_rootfs_info)



Looking at a diff from https://crosland.corp.google.com/log/9899.0.0..9900.0.0 seems like a relevant CL is:
generate_au_zip: Add unsquashfs, remove bsdiff and bspatch.
https://chromium-review.googlesource.com/#/c/chromiumos/platform/crosutils/+/644151/


Recommending that we revert this patch for now.

 
Project Member Comment 1 by bugdroid1@chromium.org, Sep 1
The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/crosutils/+/aba21ffeb9e07c54c891f012dbc9bd23d7df6dc6

commit aba21ffeb9e07c54c891f012dbc9bd23d7df6dc6
Author: Joel Kitching <kitching@chromium.org>
Date: Fri Sep 01 08:18:11 2017

Revert "generate_au_zip: Add unsquashfs, remove bsdiff and bspatch."

This reverts commit 9d29f568992107c0f537dc951a938c49190e305a.

Reason for revert: Need bspatch binary for PaygenBuild* stages.

Original change's description:
> generate_au_zip: Add unsquashfs, remove bsdiff and bspatch.
> 
> We no longer use bsdiff nor bspatch libraries to generate or apply payloads. We
> only use their shared library. This patch removes them from generate_au_zip.py.
> 
> Also we are adding squashfs update support for android container and we need our
> custom unsquashfs to generate the file map of the squashfs image. This patch
> adds it to generate_au_zip.py.
> 
> BUG=none
> TEST=submitted a tryjob
> 
> Change-Id: I027cb75321fbf26047e958a7aace90fe7afc3b4f
> Reviewed-on: https://chromium-review.googlesource.com/644151
> Commit-Ready: Amin Hassani <ahassani@chromium.org>
> Tested-by: Amin Hassani <ahassani@chromium.org>
> Reviewed-by: Dan Erat (OOO Fri-Mon) <derat@chromium.org>

BUG= 761271 

Change-Id: I4411630537d383cb3e9a88438d12a826bcdea070
Reviewed-on: https://chromium-review.googlesource.com/646812
Reviewed-by: Nicolas Boichat <drinkcat@chromium.org>
Commit-Queue: Nicolas Boichat <drinkcat@chromium.org>
Tested-by: Nicolas Boichat <drinkcat@chromium.org>

[modify] https://crrev.com/aba21ffeb9e07c54c891f012dbc9bd23d7df6dc6/build_library/generate_au_zip.py

Status: Fixed
Marking this as fixed
Sign in to add a comment