Here's a sample stacktrace from the last lulu failure:


https://uberchromegw.corp.google.com/i/chromeos/builders/lulu-release/builds/1455/steps/PaygenBuildCanary/logs/stdio


22:49:46: ERROR: <type 'exceptions.OSError'>: [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/b/c/cbuild/repository/chromite/lib/parallel.py", line 602, in TaskRunner
    task(*x, **task_kwargs)
  File "/b/c/cbuild/repository/chromite/lib/parallel.py", line 800, in <lambda>
    fn = lambda idx, task_args: out_queue.put((idx, task(*task_args)))
  File "/b/c/cbuild/repository/chromite/lib/paygen/paygen_build_lib.py", line 251, in _GenerateSinglePayload
    dry_run=dry_run)
  File "/b/c/cbuild/repository/chromite/lib/paygen/paygen_payload_lib.py", line 827, in CreateAndUploadPayload
    dry_run=dry_run).Run()
  File "/b/c/cbuild/repository/chromite/lib/paygen/paygen_payload_lib.py", line 707, in Run
    self._drm(self._VerifyPayload)
  File "/b/c/cbuild/repository/chromite/lib/paygen/dryrun_lib.py", line 45, in __call__
    return self.Run(func, *args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/paygen/dryrun_lib.py", line 82, in Run
    return self._Call(func, *args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/paygen/dryrun_lib.py", line 86, in _Call
    return func(*args, **kwargs)
  File "/b/c/cbuild/repository/chromite/lib/paygen/paygen_payload_lib.py", line 681, in _VerifyPayload
    self._ApplyPayload(payload, is_delta)
  File "/b/c/cbuild/repository/chromite/lib/paygen/paygen_payload_lib.py", line 641, in _ApplyPayload
    payload.Apply(bspatch_path=bspatch_path, **part_files)
  File "/b/c/cbuild/repository/src/platform/dev/host/lib/update_payload/payload.py", line 321, in Apply
    old_rootfs_part=old_rootfs_part)
  File "/b/c/cbuild/repository/src/platform/dev/host/lib/update_payload/applier.py", line 569, in Run
    self.payload.manifest.old_rootfs_info)
  File "/b/c/cbuild/repository/src/platform/dev/host/lib/update_payload/applier.py", line 517, in _ApplyToPartition
    new_part_file, new_part_info.size)
  File "/b/c/cbuild/repository/src/platform/dev/host/lib/update_payload/applier.py", line 468, in _ApplyOperations
    new_part_file)
  File "/b/c/cbuild/repository/src/platform/dev/host/lib/update_payload/applier.py", line 389, in _ApplySourceBsdiffOperation
    subprocess.check_call(bspatch_cmd)
  File "/usr/lib/python2.7/subprocess.py", line 535, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 522, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory


applier.py:387-389:
      bspatch_cmd = [self.bspatch_path, old_file_name, new_file_name,
                     patch_file_name, in_extents_arg, out_extents_arg]
      subprocess.check_call(bspatch_cmd)


Going a little up the stack, there is a "paygen_build_lib.py" being kicked off by some task queue.  This file seems to call into paygen_payload_lib.CreateAndUploadPayload, whose first step is to download and unzip this "au-generator.zip", putting its binaries in PATH.


Then we jump from paygen_*_lib into upload_payload/applier.py, which uses bspatch (which looks for it in generator_dir, which is where au-generator.zip was extracted):
  File "/b/c/cbuild/repository/chromite/lib/paygen/paygen_payload_lib.py", line 641, in _ApplyPayload
    bspatch_path = os.path.join(self.generator_dir, 'bspatch')
    try:
      payload.Apply(bspatch_path=bspatch_path, **part_files)
  File "/b/c/cbuild/repository/src/platform/dev/host/lib/update_payload/payload.py", line 321, in Apply
    old_rootfs_part=old_rootfs_part)
  File "/b/c/cbuild/repository/src/platform/dev/host/lib/update_payload/applier.py", line 569, in Run
    self.payload.manifest.old_rootfs_info)



Looking at a diff from https://crosland.corp.google.com/log/9899.0.0..9900.0.0 seems like a relevant CL is:
generate_au_zip: Add unsquashfs, remove bsdiff and bspatch.
https://chromium-review.googlesource.com/#/c/chromiumos/platform/crosutils/+/644151/


Recommending that we revert this patch for now.