Last night's Chrome PFQ run failed due to a provisioning error in the veyron_minnie-chrome-pfq run at http://cros-goldeneye/chromeos/healthmonitoring/buildDetails?buildbucketId=8928452740594651024:
"provision: FAIL: Download and install failed from chromeos4-devserver9.cros.corp.google.com onto chromeos4-row9-rack9-host21: command execution error"
In the logs at http://stainless/browse/chromeos-autotest-results/262496262-chromeos-test/, I see the following in https://storage.cloud.google.com/chromeos-autotest-results/262496262-chromeos-test/chromeos4-row9-rack9-host21/debug/autoserv.ERROR:
...
Set boot target to /dev/mmcblk0p3: Partition 3, Slot A
SetImage
KERNEL_CONFIG: console= loglevel=7 init=/sbin/init cros_secure oops=panic panic=-1 root=/dev/dm-0 rootwait ro dm_verity.error_behavior=3 dm_verity.max_bios=-1 dm_verity.dev_wait=1 dm="1 vroot none ro 1,0 3334144 verity payload=PARTUUID=%U/PARTNROFF=1 hashtree=PARTUUID=%U/PARTNROFF=1 hashstart=3334144 alg=sha1 root_hexdigest=b29b7b27188d6bc42b4cfa6ca3d0376c2e5858af salt=97cadae5fc23783744fd49aef618c4696bdf82316c723475fe4e264376951896" noinitrd cros_debug vt.global_cursor_default=0 kern_guid=%U
Setting up verity.
Finished after 53 seconds.
SetImage failed.
PostInstall Failed
Downloaded hashes prior to postinst failure:
d4597ea7803d2893e574f0e1371bcfa3 -
8bbba3999f9a245071d40410f2a40a4f -
5d99a1239331daf75b5e9dd57bc4681a -
2018-11-30 01:13:56-08:00 ERROR: FATAL: postinst failed.
2018-11-30 01:13:56-08:00 INFO: Updated status: FATAL: postinst failed.
11/30 01:16:09.823 ERROR| autoupdater:1006| Failure during download and install.
...
Setting up verity.
Finished after 65 seconds.
SetImage failed.
PostInstall Failed
...
I think that this may be the relevant error (from the same file):
2018-11-30 01:13:02-08:00 INFO: Update next kernel to try (via postinst)
dm:dm bht[DEBUG] Setting block_count 416768
dm:dm bht[DEBUG] Setting depth to 3.
dm:dm bht[DEBUG] depth: 0 entries: 1
dm:dm bht[DEBUG] depth: 1 entries: 26
dm:dm bht[DEBUG] depth: 2 entries: 3256
[1130/011356:ERROR:chromeos_verity.cc(280)] Filesystem hash verification failed
[1130/011356:ERROR:chromeos_verity.cc(281)] Expected b29b7b27188d6bc42b4cfa6ca3d0376c2e5858af != actual c6e624b23c2816295f318ca25080c67f8dee8319
PostInstall Configured: (A, /dev/mmcblk0p3, /dev/mmcblk0p2, /dev/mmcblk0p12)
Issue 899583 looks like it was an occurrence of the same error on tricky-chrome-pfq from a month ago, but it was archived without any resolution being stated.
Allen (current deputy), please investigate this to figure out what happened and how it can be prevented. Would a retry have helped? Do we need to verify checksums at some earlier step?
Comment 1 by ayatane@chromium.org
, Nov 30