Another one (for comparison):

https://uberchromegw.corp.google.com/i/chromeos/builders/cave-release/builds/753/steps/Archive/logs/stdio


13:35:23: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -m cp -v -- /b/cbuild/internal_master/buildbot_archive/cave-release/R57-9178.0.0/stripped-packages.tar gs://chromeos-moblab-asus/cave-release/R57-9178.0.0/stripped-packages.tar
13:35:42: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -m cp -v -- /b/cbuild/internal_master/buildbot_archive/cave-release/R57-9178.0.0/stripped-packages.tar gs://chromeos-moblab-intel/cave-release/R57-9178.0.0/stripped-packages.tar
13:35:55: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -m cp -v -- /b/cbuild/internal_master/buildbot_archive/cave-release/R57-9178.0.0/UPLOADED gs://chromeos-image-archive/cave-release/R57-9178.0.0/UPLOADED
13:36:00: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -m cp -v -- /b/cbuild/internal_master/buildbot_archive/cave-release/R57-9178.0.0/UPLOADED gs://chromeos-moblab-asus/cave-release/R57-9178.0.0/UPLOADED
13:36:04: INFO: RunCommand: /b/cbuild/internal_master/.cache/common/gsutil_4.19.tar.gz/gsutil/gsutil -o 'Boto:num_retries=10' -m cp -v -- /b/cbuild/internal_master/buildbot_archive/cave-release/R57-9178.0.0/UPLOADED gs://chromeos-moblab-intel/cave-release/R57-9178.0.0/UPLOADED

@@@STEP_FAILURE@@@
13:57:27: ERROR: <class 'chromite.lib.failures_lib.BuildScriptFailure'>: ./build_image failed (code=1)
Traceback (most recent call last):
  File "/b/cbuild/internal_master/chromite/lib/parallel.py", line 441, in _Run
    self._task(*self._task_args, **self._task_kwargs)
  File "/b/cbuild/internal_master/chromite/cbuildbot/stages/artifact_stages.py", line 205, in BuildAndArchiveFactoryImages
    alias = commands.BuildFactoryInstallImage(buildroot, board, extra_env)
  File "/b/cbuild/internal_master/chromite/cbuildbot/commands.py", line 1928, in BuildFactoryInstallImage
    enter_chroot=True)
  File "/b/cbuild/internal_master/chromite/cbuildbot/commands.py", line 139, in RunBuildScript
    raise failures_lib.BuildScriptFailure(ex, cmd[0])
  File "/b/cbuild/internal_master/chromite/cbuildbot/commands.py", line 139, in RunBuildScript
    raise failures_lib.BuildScriptFailure(ex, cmd[0])
BuildScriptFailure: ./build_image failed (code=1)

Now basically all canary bots are failing in the "Archive" step.

https://uberchromegw.corp.google.com/i/chromeos/waterfall?builder=master-release&builder=arkham-release&builder=asuka-release&builder=auron_paine-release&builder=auron_yuna-release&builder=banjo-release&builder=banon-release&builder=beaglebone-release&builder=beaglebone_servo-release&builder=buddy-release&builder=butterfly-release&builder=candy-release&builder=caroline-release&builder=cave-release&builder=celes-release&builder=clapper-release&builder=cyan-release&builder=daisy-release&builder=daisy_skate-release&builder=daisy_spring-release&builder=edgar-release&builder=elm-release&builder=enguarde-release&builder=expresso-release&builder=falco-release&builder=falco_li-release&builder=gale-release&builder=gandof-release&builder=glimmer-release&builder=gnawty-release&builder=gru-release&builder=guado-release&builder=heli-release&builder=jecht-release&builder=kefka-release&builder=kevin-release&builder=kip-release&builder=lakitu-release&builder=lars-release&builder=leon-release&builder=link-release&builder=lulu-release&builder=lumpy-release&builder=mccloud-release&builder=monroe-release&builder=ninja-release&builder=nyan_big-release&builder=nyan_blaze-release&builder=nyan_kitty-release&builder=oak-release&builder=orco-release&builder=panther-release&builder=parrot-release&builder=parrot_ivb-release&builder=peach_pi-release&builder=peach_pit-release&builder=peppy-release&builder=quawks-release&builder=reks-release&builder=relm-release&builder=rikku-release&builder=samus-release&builder=sentry-release&builder=setzer-release&builder=squawks-release&builder=stout-release&builder=stumpy-release&builder=sumo-release&builder=swanky-release&builder=terra-release&builder=tidus-release&builder=tricky-release&builder=ultima-release&builder=umaro-release&builder=veyron_jaq-release&builder=veyron_jerry-release&builder=veyron_mighty-release&builder=veyron_minnie-release&builder=veyron_rialto-release&builder=veyron_speedy-release&builder=whirlwind-release&builder=winky-release&builder=wizpig-release&builder=wolf-release&builder=x86-alex-release&builder=x86-alex_he-release&builder=x86-zgb-release&builder=x86-zgb_he-release&builder=zako-release&titles=off&reload=30

Presumably between 9176 and 9177
https://crosland.corp.google.com/log/9176.0.0..9177.0.0

https://uberchromegw.corp.google.com/i/chromeos/builders/samus-release
https://uberchromegw.corp.google.com/i/chromeos/builders/auron_paine-release
https://uberchromegw.corp.google.com/i/chromeos/builders/cave-release

After 9179 (https://crosland.corp.google.com/log/9178.0.0..9179.0.0) we see some how very clear error.

ERROR   : Thu Jan 12 22:24:44 PST 2017
ERROR   :  PGID  PPID   PID     ELAPSED     TIME %CPU COMMAND
ERROR   : Arguments of 10: ./mod_image_for_recovery.sh '--board=cave' '--image=/mnt/host/source/src/build/images/cave/R57-9179.0.0/tmp8P8iEc/chromiumos_base_image.bin'
ERROR   : Backtrace:  (most recent call is last)
ERROR   :  mod_image_for_recovery.sh:385:main(), called: emerge_custom_kernel '/build/cave/factory-root' 
ERROR   :  common.sh:1240:emerge_custom_kernel(), called: die 'Cannot emerge chromeos-initramfs' 
ERROR   : 
ERROR   : Error was:
ERROR   :   Cannot emerge chromeos-initramfs


For 9177 and 9178 (https://crosland.corp.google.com/log/9176.0.0..9177.0.0),

to me, the log is vague and cannot tell immediately what is failing.
The last command is indeed gsutil but I'm not sure if it is the blame.

One suspicious line I see is

> mkdir: cannot create directory ‘/mnt/host/source/src/build/images/cave/R57-9178.0.0-a3/rootfs/opt/google/containers’: Permission denied

FYI, re: 9178..9179, GCE builder reimage happened between those revisions.
https://groups.google.com/a/google.com/d/msg/chromeos-team/zo__lby2xTs/glFWbmOXEgAJ

For the mod_image_for_recovery.sh failure, its dying message is

lddtree: warning: /build/cave/sbin/udevadm: did not match any paths
make[1]: *** [../common/initramfs.mk:50: stage_init] Error 1

mka@ vapier@, is this something related to the udev change https://chromium-review.googlesource.com/#/c/409017/ ?

lhchavez@ and ARC constables, my super wild guess on "mkdir /opt/google/containers" failure originates from:
https://chrome-internal-review.googlesource.com/#/c/314713/
Can it be making the problem or is it just a red herring?

 Issue 681001  has been merged into this issue.

Kazuhiro: your super-wild theory is intriguing.

You may want to join us at https://comlink.googleplex.com/crosoncall for additional communication.  (Yes I know, the timing doesn't work, but there's still useful information.)

Thanks!

P.S.  I tried reproducing the emerge chromeos-initramfs error, but it works fine on my workstation.  I used 

USE=recovery_ramfs emerge-cyan chromeos-initramfs

re #6:

The udev-208 creates a symlink /sbin/udevadm => /bin/udevadm, udev-225 does not. Should we add the symlink or adapt mod_image_for_recovery.sh?

CL being tested by author at https://chromium-review.googlesource.com/#/c/427900/, then we plan to chump it.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/3a939f60dca422a57592cca531b62249c19e9c5c

commit 3a939f60dca422a57592cca531b62249c19e9c5c
Author: Mike Frysinger <vapier@chromium.org>
Date: Fri Jan 13 19:04:21 2017

chromeos-initramfs: update path to udevadm

Newer udevs install into the /bin/ subdir.

BUG= chromium:680849 
TEST=precq passes

Change-Id: I8baa0a9c547dde4bf7cd1ad6c38e23ded9f5ec75
Reviewed-on: https://chromium-review.googlesource.com/427900
Reviewed-by: Luigi Semenzato <semenzato@chromium.org>
Reviewed-by: Matthias Kaehlcke <mka@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/3a939f60dca422a57592cca531b62249c19e9c5c/chromeos-base/chromeos-initramfs/chromeos-initramfs-9999.ebuild

should be fixed now

Let's call this one fixed, and move the work for the unknown build_image failure (which is still happening) to  issue 680532 .