host: chromeos4-row12-rack11-host1, status: Repair Failed, locked: False diagnosis: Failed repair

Looks to be passing once in a while, so dropping this down to a p1. We have too many P0s open right now.

Given we haven't had a successful PFQ run since 9/6 I really think everything standing in the way of that needs to stay a P0.

Is there a second known instance of this? If so, please link to it. If not or absent other evidence that it is then it might not be standing in the way of uprev and hence not P0.

Also, I see on crosoncall that chrome was just now manually upreved. Therefore, demoting to P1.

ImageInstallError: Download and install failed from chromeos4-devserver4.cros.corp.google.com onto chromeos4-row11-rack10-host15: command execution error
* Command: 
    /usr/bin/ssh -a -x  -o ControlPath=/tmp/_autotmp_l7Va2Wssh-master/socket
    -o Protocol=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
    -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o
    ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22
    chromeos4-row11-rack10-host15 "export LIBC_FATAL_STDERR_=1; if type
    \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\"
    \"server[stack::_revert_boot_partition|_run|run] -> ssh_run(/postinst
    /dev/mmcblk0p3 2>&1)\";fi; /postinst /dev/mmcblk0p3 2>&1"
Exit status: 1
Duration: 24.8934381008

stdout:
dm:dm bht[DEBUG] Setting block_count 448000
dm:dm bht[DEBUG] Setting depth to 3.
dm:dm bht[DEBUG] depth: 0 entries: 1
dm:dm bht[DEBUG] depth: 1 entries: 28
dm:dm bht[DEBUG] depth: 2 entries: 3500
PostInstall Configured: (A, /dev/mmcblk0p3, /dev/mmcblk0p2, /dev/mmcblk0p12)

 Current Kernel Info: sysname(Linux) nodename(localhost) release(3.18.0-18125-ge005298cc868) version(#1 SMP PREEMPT Thu Sep 13 00:19:56 PDT 2018) machine(x86_64)

lsb-release inside the new rootfs:
CHROMEOS_RELEASE_APPID={11130F0B-738A-C024-7A78-CF72D93B77AF}
CHROMEOS_BOARD_APPID={11130F0B-738A-C024-7A78-CF72D93B77AF}
CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1}
DEVICETYPE=CHROMEBOOK
CHROMEOS_ARC_VERSION=5006116
CHROMEOS_ARC_ANDROID_SDK_VERSION=25
GOOGLE_RELEASE=10895.56.0
CHROMEOS_DEVSERVER=
CHROMEOS_RELEASE_BUILDER_PATH=cyan-release/R69-10895.56.0
CHROMEOS_RELEASE_BUILD_NUMBER=10895
CHROMEOS_RELEASE_BRANCH_NUMBER=56
CHROMEOS_RELEASE_CHROME_MILESTONE=69
CHROMEOS_RELEASE_PATCH_NUMBER=0
CHROMEOS_RELEASE_TRACK=testimage-channel
CHROMEOS_RELEASE_DESCRIPTION=10895.56.0 (Official Build) dev-channel cyan test
CHROMEOS_RELEASE_BUILD_TYPE=Official Build
CHROMEOS_RELEASE_NAME=Chrome OS
CHROMEOS_RELEASE_BOARD=cyan
CHROMEOS_RELEASE_VERSION=10895.56.0
CHROMEOS_AUSERVER=https://tools.google.com/service/update2

Set boot target to /dev/mmcblk0p3: Partition 3, Slot A
SetImage
KERNEL_CONFIG: console= loglevel=7 init=/sbin/init cros_secure oops=panic panic=-1 root=/dev/dm-0 rootwait ro dm_verity.error_behavior=3 dm_verity.max_bios=-1 dm_verity.dev_wait=1 dm="1 vroot none ro 1,0 3584000 verity payload=PARTUUID=%U/PARTNROFF=1 hashtree=PARTUUID=%U/PARTNROFF=1 hashstart=3584000 alg=sha1 root_hexdigest=7461950aec26f9cce6f8f05c4541424720341083 salt=310157aa274deeeb2f115190a4dd6bd4a4aa0e43d144c99fdf610a34569d2942" noinitrd cros_debug vt.global_cursor_default=0 kern_guid=%U add_efi_memmap boot=local noresume noswap i915.modeset=1 tpm_tis.force=1 tpm_tis.interrupts=0 nmi_watchdog=panic,lapic iwlwifi.remove_when_gone=1 
Setting up verity.
Filesystem hash verification failed
Expected 7461950aec26f9cce6f8f05c4541424720341083 != actual 70d233b3ab31756943762c9f1ad2447901f2015d
Finished after 24 seconds.
SetImage failed.
PostInstall Failed

Looks like we have a second instance of this one, that's two pfq runs in a row that have been taken out by cyan-chrome-pfq. Once this is fixed we will have a green PFQ!!

https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8935394395256094352

Same dut, devserver implicated

chromeos4-devserver6.cros.corp.google.com onto chromeos4-row11-rack10-host15

  Filesystem hash verification failed
  Expected 7461950aec26f9cce6f8f05c4541424720341083 != actual bb8e78059df1616bed9bede2193c3f17f75b8e99
  

Looks like that device may actually be bad, maybe I should go buy a lottery ticket.

Going to see if I can swap out that dut

locked the dut, hopefully cyan will pass now

cyan PFQ passed, that DUT was bad

Files separate ticket for DUT followup