It sounds like the payload it tried to download was invalid.

I don't know enough about moblab payload/recovery image publication to diagnose.

Moblab AU should just be exactly the same as any other ChromeOS image, the only difference being we never serve deltas

It looks like this may be root cause

[0425/161501:INFO:delta_performer.cc(1343)] Verifying metadata hash signature using public key: /usr/share/update_engine/update-payload-key.pub.pem
[0425/161501:ERROR:payload_verifier.cc(143)] Unable to open public key file: /usr/share/update_engine/update-payload-key.pub.pem
[0425/161501:ERROR:delta_performer.cc(1365)] Unable to compute expected hash from metadata signature

That file doesn't exist on the moblab

It could be that it doesn't have access to that file due to some disk IO problem.
That file should basically exist on all devices. I don't think the existence of that file is an issue. Someone with moblab experience should check that though.

I do not have access to a 63 image right at the moment but for a 65 image that file seem to be missing :

moblab@localhost /usr/share $ sudo ls -lrtd u*                                                                          
drwxr-xr-x 4 root root 4096 Apr 25 13:36 userfeedback

Moblabs are running in dev mode and USE="-coreboot -encrypted_stateful" is set - however up to 63 we have been able to AU as regular chromeos devices have

Confirming for R63-10032.89.0 the file /usr/share/update_engine/update-payload-key.pub.pem does not exist. The /usr/share/update_engine directory does not exist either.

Current working theory is that this hack has broken something:

https://cs.corp.google.com/chromeos_public/src/third_party/chromiumos-overlay/chromeos-base/update_payload/update_payload-9999.ebuild?rcl=6880f112b1aa0323377098dfb52f09e8a0c6eccf&l=31

So on moblab the pem file ends up in /usr/lib64/python2.7/site-packages/update_payload

I posted  https://chromium-review.googlesource.com/#/c/chromiumos/overlays/chromiumos-overlay/+/1030714 as an example of what I want to achieve but understand there is likely to be better ways to achieve that goal.

Also how to test something like this without submitting ?

I believe that if you just do a tryjob that generates the image, you can check for the presence of the .pem file.

However, it's (intentionally) hard to get signed images/payloads for a build that includes unsubmitted code.

65 moblab needs to go out upping to p0.

Unrelated devserver changes have resulted in moblab not being able to provision DUT's with 67/68 images.

This is blocking partner testing for a device which only has 68 builds.

It seems (deliberately) hard to test this with un-submitted code.  The proposed CL is clearly not the right way to achieve the desired outcome.

My current plan is to check in a copy of the pem file in the project-moblab overlay and install it in the desired location just for moblab.  I can check in that CL for testing on 65 and revert it later when the correct solution is found.  I want to make sure that this missing pem file is the root cause of this bug and if the file needs to be on 65 image or 63 image.  I will not check in on ToT

Adding Prathmesh in case he has any other ideas on how to fix this.

If we have to hack this for moblab for now, I'd just add a symlink to where the key file is ending up, in the moblab overlay. (instead of copying the file to the overlay)

I'm not familiar with moblab, apologies if I'm missing something.

iiuc, /usr/lib64/python2.7/site-packages/update_payload is for the devserver running on moblab. It shouldn't be related to the updates for moblab itself, correct? So https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/614389 shouldn't have an impact on /usr/share/update_engine/update-payload-key.pub.pem, right?

Where is the R63-10032.89.0 image coming from? I see one in the beta-channel https://pantheon.corp.google.com/storage/browser/chromeos-releases/beta-channel/guado-moblab/10032.89.0/?pli=1, but according to the update_engine logs it seems to think that it's on the dev channel, transitioning to the beta channel?

I checked the content of chromeos_10032.89.0_guado-moblab_recovery_beta-channel_premp.bin, it looks like /usr/share/update_engine/update-payload-key.pub.pem is present, I don't understand why it's not on your device. Is it a different image?

https://pantheon.corp.google.com/storage/browser/chromeos-releases/stable-channel/guado-moblab/10032.89.0 should be the release image that partners are running and is pushed live.

What ebuild would install  /usr/share/update_engine/update-payload-key.pub.pem - from my testing 

https://cs.corp.google.com/chromeos_public/src/third_party/chromiumos-overlay/chromeos-base/update_payload/update_payload-9999.ebuild?type=cs&q=update-payload-key.pub.pem+file:.*ebuild&g=0&l=38

does not (it gets installed in /usr/lib64/python2.7/site-packages/update_payload)

It is entirely possible I have gone completely wrong in this investigation this was the only ebuild I could find that mentioned the pem file

Right, it looks like the recovery does have the public key:

(cr) ((95277f5...)) norvez@norvez ~/trunk/src/scripts $ ./mount_gpt_image.sh --read_only -f /tmp/chromeos_10032.89.0_guado-moblab_re
covery_stable-channel_premp.bin 
mount: special device /tmp/s/var_overlay does not exist
(cr) ((95277f5...)) norvez@norvez ~/trunk/src/scripts $ cat /tmp/m/etc/lsb-release 
CHROMEOS_AUSERVER=https://tools.google.com/service/update2
CHROMEOS_BOARD_APPID={BCC6E30A-D921-18E8-73D3-60177AD4AF86}
CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1}
CHROMEOS_DEVSERVER=
CHROMEOS_RELEASE_APPID={BCC6E30A-D921-18E8-73D3-60177AD4AF86}
CHROMEOS_RELEASE_BOARD=guado_moblab-signed-moblab-prempkeys
CHROMEOS_RELEASE_BRANCH_NUMBER=89
CHROMEOS_RELEASE_BUILDER_PATH=guado_moblab-release/R63-10032.89.0
CHROMEOS_RELEASE_BUILD_NUMBER=10032
CHROMEOS_RELEASE_BUILD_TYPE=Official Build
CHROMEOS_RELEASE_CHROME_MILESTONE=63
CHROMEOS_RELEASE_DESCRIPTION=10032.89.0 (Official Build) stable-channel guado_moblab 
CHROMEOS_RELEASE_NAME=Chrome OS
CHROMEOS_RELEASE_PATCH_NUMBER=0
CHROMEOS_RELEASE_TRACK=stable-channel
CHROMEOS_RELEASE_VERSION=10032.89.0
DEVICETYPE=CHROMEBOX
GOOGLE_RELEASE=10032.89.0
(cr) ((95277f5...)) norvez@norvez ~/trunk/src/scripts $ cat /tmp/m/usr/share/update_engine/update-payload-key.pub.pem 
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1Bg9BnjWhX3jJyECeXqF
O28nkYTF1NHWLlFHgzAGg+ysva22BL3S5LlsNejnYVg/xzx3izvAQyOF3I1TJVOy
2fH1DoZOWyKuckMyUrFQbO6OV1VIvPUPKckHadWcXSsHj2lBdDPH9xRDEBsXeztf
nAGBD8GlAyTU7iH+Bf+xzyK9k4BmITf4Nx4xWhRZ6gm2Fc2SEP3x5N5fohkLv5ZP
kFr0fj5wUK+0XF95rkGFBLIq2XACS3dmxMFToFl1HMM1HonUg9TAH+3dVH93zue1
y81mkTuGnNX+zYya5ov2kD8zW1V10iTOSJfOlho5T8FpKbG37o3yYcUiyMHKO1Iv
PQIDAQAB
-----END PUBLIC KEY-----
(cr) ((95277f5...)) norvez@norvez ~/trunk/src/scripts $

My understanding is that the public key is not installed by an ebuild but by the signers:

https://cs.corp.google.com/chromeos_public/src/platform/vboot_reference/scripts/image_signing/insert_au_publickey.sh?rcl=8c34ae60786f7ff28b24ceb1b065a3b42e63c3c9&l=28

https://cs.corp.google.com/chromeos_internal/cros-signing/signer/image_signer.py?rcl=61b863278e0df724c6340e772fb4e13a2ebfc0f0&l=740

Thanks for the info - we are re-doing the test to see if we have made a mistake somewhere in the setup.

I just tried again using this image
https://storage.cloud.google.com/chromeos-releases/stable-channel/guado-moblab/10032.89.0/ChromeOS-recovery-R63-10032.89.0-guado-moblab.tar.xz?_ga=2.1772849.-1003776878.1517247667

After running the recovery, I first checked if the key is there. It was not there, and the /usr/share/update_engine directory is not there.

I also tried to run an update to R65 after switching the device to beta channel and got the same DownloadMetadataSignatureError

I think that's the wrong image.

Can you try https://storage.cloud.google.com/chromeos-releases/stable-channel/guado-moblab/10032.89.0/chromeos_10032.89.0_guado-moblab_recovery_stable-channel_premp.bin.zip?_ga=2.221079320.-1622923230.1524780818 ?

/etc/lsb-release of the recovery image you used so far:

HROMEOS_RELEASE_APPID={BCC6E30A-D921-18E8-73D3-60177AD4AF86}
CHROMEOS_BOARD_APPID={BCC6E30A-D921-18E8-73D3-60177AD4AF86}
CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1}
DEVICETYPE=CHROMEBOX
CHROMEOS_RELEASE_BUILDER_PATH=guado_moblab-release/R63-10032.89.0
GOOGLE_RELEASE=10032.89.0
CHROMEOS_DEVSERVER=
CHROMEOS_RELEASE_BOARD=guado_moblab
CHROMEOS_RELEASE_BUILD_NUMBER=10032
CHROMEOS_RELEASE_BRANCH_NUMBER=89
CHROMEOS_RELEASE_CHROME_MILESTONE=63
CHROMEOS_RELEASE_PATCH_NUMBER=0
CHROMEOS_RELEASE_TRACK=dev-channel
CHROMEOS_RELEASE_DESCRIPTION=10032.89.0 (Official Build) dev-channel guado_moblab test
CHROMEOS_RELEASE_BUILD_TYPE=Official Build
CHROMEOS_RELEASE_NAME=Chrome OS
CHROMEOS_RELEASE_VERSION=10032.89.0
CHROMEOS_AUSERVER=https://tools.google.com/service/update2


That doesn't look like an AU-capable image, that's probably the issue.

Looks like the key is there on that image - looks like this was a testing instruction mistake, so reducing priority but I will double check 

The recovery USB was originally created with the command:

cros flash usb:// xbuddy://remote/guado_moblab-release/R63-10032.89.0/recovery

But it is looking like that serves up the other recovery image.

Ok - testing error - sorry for the firedrill