samus-android-pfq: provisioning failures caused by 'read-only' partition |
|||||||||||||
Issue descriptionSamus Chrome PFQ builds https://uberchromegw.corp.google.com/i/chromeos/builders/samus-android-pfq/builds/210 https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/74863066-chromeos-test/chromeos2-row4-rack1-host9/provision_AutoUpdate/ 08/28 19:21:18.599 ERROR| base_utils:0280| [stderr] touch: cannot touch '/var/tmp/provision_failed': Read-only file system and https://uberchromegw.corp.google.com/i/chromeos/builders/samus-android-pfq/builds/211 https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/74893837-chromeos-test/chromeos2-row4-rack3-host9/provision_AutoUpdate/debug/ 8/28 23:23:20.839 DEBUG| base_utils:0185| Running 'rsync -L --delete --timeout=1800 --rsh='/usr/bin/ssh -a -x -o ControlPath=/tmp/_autotmp_63vFq9ssh-master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22' -az --no-o --no-g "/usr/local/google/chromeos/src/platform/dev/stateful_update" "root@chromeos2-row4-rack3-host9:"/tmp/stateful_update""' 08/28 23:23:21.346 DEBUG| ssh_host:0180| Running (ssh) '/tmp/stateful_update --stateful_change=reset 2>&1' 08/28 23:23:21.593 DEBUG| base_utils:0280| [stdout] Resetting stateful update state. 08/28 23:23:21.595 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/iproute2-3.8.0-r1': Input/output error 08/28 23:23:21.595 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/DEFINED_PHASES': Input/output error 08/28 23:23:21.596 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/environment.bz2': Input/output error 08/28 23:23:21.596 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/IUSE': Input/output error 08/28 23:23:21.596 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/repository': Input/output error 08/28 23:23:21.597 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/USE': Input/output error 08/28 23:23:21.597 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/PF': Input/output error 08/28 23:23:21.598 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/CATEGORY': Input/output error 08/28 23:23:21.598 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/BUILD_TIME': Input/output error 08/28 23:23:21.598 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/CBUILD': Input/output error 08/28 23:23:21.598 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/CXXFLAGS': Input/output error 08/28 23:23:21.599 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/findutils-4.4.2-r2.ebuild': Input/output error 08/28 23:23:21.599 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/CONTENTS': Input/output error 08/28 23:23:21.599 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/CHOST': Input/output error 08/28 23:23:21.600 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/NEEDED.ELF.2': Input/output error 08/28 23:23:21.600 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/SLOT': Input/output error 08/28 23:23:21.600 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/NEEDED': Input/output error 08/28 23:23:21.600 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/EAPI': Input/output error 08/28 23:23:21.601 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/SIZE': Input/output error 08/28 23:23:21.601 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/DESCRIPTION': Input/output error 08/28 23:23:21.601 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/LDFLAGS': Input/output error 08/28 23:23:21.602 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/KEYWORDS': Input/output error 08/28 23:23:21.602 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/license.yaml': Input/output error 08/28 23:23:21.602 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/CFLAGS': Input/output error 08/28 23:23:21.603 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/BINPKGMD5': Input/output error 08/28 23:23:21.603 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/LICENSE': Input/output error 08/28 23:23:21.603 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/HOMEPAGE': Input/output error 08/28 23:23:21.604 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/INHERITED': Input/output error 08/28 23:23:21.604 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/findutils-4.4.2-r2/COUNTER': Input/output error 08/28 23:23:21.604 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/DEFINED_PHASES': Input/output error 08/28 23:23:21.604 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/environment.bz2': Input/output error 08/28 23:23:21.605 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/RDEPEND': Input/output error 08/28 23:23:21.605 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/IUSE': Input/output error 08/28 23:23:21.605 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/repository': Input/output error 08/28 23:23:21.606 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/USE': Input/output error 08/28 23:23:21.606 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/PF': Input/output error 08/28 23:23:21.606 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/CATEGORY': Input/output error 08/28 23:23:21.606 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/BUILD_TIME': Input/output error 08/28 23:23:21.607 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/FEATURES': Input/output error 08/28 23:23:21.607 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/CBUILD': Input/output error 08/28 23:23:21.607 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/CXXFLAGS': Input/output error 08/28 23:23:21.608 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/mosys-1.2.03-r290.ebuild': Input/output error 08/28 23:23:21.608 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/CONTENTS': Input/output error 08/28 23:23:21.608 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/CHOST': Input/output error 08/28 23:23:21.608 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/SLOT': Input/output error 08/28 23:23:21.609 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/EAPI': Input/output error 08/28 23:23:21.609 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/SIZE': Input/output error 08/28 23:23:21.609 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/DESCRIPTION': Input/output error 08/28 23:23:21.609 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/LDFLAGS': Input/output error 08/28 23:23:21.610 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/KEYWORDS': Input/output error 08/28 23:23:21.610 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/license.yaml': Input/output error 08/28 23:23:21.610 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/CFLAGS': Input/output error 08/28 23:23:21.611 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/BINPKGMD5': Input/output error 08/28 23:23:21.611 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/LICENSE': Input/output error 08/28 23:23:21.611 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/HOMEPAGE': Input/output error 08/28 23:23:21.611 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/CC': Input/output error 08/28 23:23:21.612 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/INHERITED': Input/output error 08/28 23:23:21.612 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/DEPEND': Input/output error 08/28 23:23:21.612 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/mosys-1.2.03-r290/COUNTER': Input/output error 08/28 23:23:21.613 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/net-tools-1.60_p20130513023548-r1': Input/output error 08/28 23:23:21.613 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/DEFINED_PHASES': Input/output error 08/28 23:23:21.613 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/usbutils-007.ebuild': Input/output error 08/28 23:23:21.613 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/environment.bz2': Input/output error 08/28 23:23:21.614 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/RDEPEND': Input/output error 08/28 23:23:21.614 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/IUSE': Input/output error 08/28 23:23:21.614 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/repository': Input/output error 08/28 23:23:21.614 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/USE': Input/output error 08/28 23:23:21.615 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/PF': Input/output error 08/28 23:23:21.615 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/CATEGORY': Input/output error 08/28 23:23:21.615 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/BUILD_TIME': Input/output error 08/28 23:23:21.616 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/IUSE_EFFECTIVE': Input/output error 08/28 23:23:21.616 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/CBUILD': Input/output error 08/28 23:23:21.616 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/CXXFLAGS': Input/output error 08/28 23:23:21.617 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/CONTENTS': Input/output error 08/28 23:23:21.617 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/CHOST': Input/output error 08/28 23:23:21.617 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/NEEDED.ELF.2': Input/output error 08/28 23:23:21.618 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/SLOT': Input/output error 08/28 23:23:21.618 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/NEEDED': Input/output error 08/28 23:23:21.618 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/EAPI': Input/output error 08/28 23:23:21.618 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/SIZE': Input/output error 08/28 23:23:21.630 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/DESCRIPTION': Input/output error 08/28 23:23:21.631 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/LDFLAGS': Input/output error 08/28 23:23:21.631 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/license.yaml': Input/output error 08/28 23:23:21.631 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/CFLAGS': Input/output error 08/28 23:23:21.632 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/BINPKGMD5': Input/output error 08/28 23:23:21.632 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/LICENSE': Input/output error 08/28 23:23:21.632 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/HOMEPAGE': Input/output error 08/28 23:23:21.632 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/INHERITED': Input/output error 08/28 23:23:21.633 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/DEPEND': Input/output error 08/28 23:23:21.633 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/usbutils-007/COUNTER': Input/output error 08/28 23:23:21.633 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/sandbox-2.6-r1': Input/output error 08/28 23:23:21.634 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/ethtool-3.12.1': Input/output error 08/28 23:23:21.634 DEBUG| base_utils:0280| [stdout] rm: cannot remove '/mnt/stateful_partition/var_new/db/pkg/sys-apps/lshw-02.17b-r1': Input/output error 08/28 23:23:21.635 WARNI| cros_host:0825| Autoupdate did not complete. 08/28 23:23:21.645 WARNI| test:0606| Autotest caught exception when running test: Traceback (most recent call last): File "/usr/local/autotest/client/common_lib/test.py", line 600, in _exec _call_test_function(self.execute, *p_args, **p_dargs) File "/usr/local/autotest/client/common_lib/test.py", line 810, in _call_test_function raise error.UnhandledTestFail(e) UnhandledTestFail: Unhandled AutoservRunError: command execution error * Command: /usr/bin/ssh -a -x -o ControlPath=/tmp/_autotmp_63vFq9ssh- master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22 chromeos2-row4-rack3-host9 "export LIBC_FATAL_STDERR_=1; if type \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\" \"server[stack::run_update|reset_stateful_partition|_run] -> ssh_run(/tmp/stateful_update --stateful_change=reset 2>&1)\";fi; /tmp/stateful_update --stateful_change=reset 2>&1" Exit status: 1 Duration: 0.181783914566
,
Aug 29 2016
Raise to P1 since it's blocking Android PFQ.
,
Aug 29 2016
(Looks like we still uprev even though builders are red)
,
Aug 29 2016
So far the errors all happened on chromeos2-row4-rack3-host9. Let me check with the deputy...
,
Aug 29 2016
nxia@ will help look into the DUT.
,
Aug 29 2016
xixuan@ has locked that dut. Will file a ticket to fix that dut.
,
Aug 29 2016
this pfq builder is back to green. filed the ticket at https://b.corp.google.com/u/0/issues/31156292.
,
Sep 2 2016
,
Oct 7 2016
,
Oct 10 2016
,
Nov 19 2016
,
Jan 21 2017
,
Mar 4 2017
,
Apr 17 2017
,
May 30 2017
,
Aug 1 2017
,
Oct 14 2017
|
|||||||||||||
►
Sign in to add a comment |
|||||||||||||
Comment 1 by cywang@chromium.org
, Aug 29 2016