moblab verify should include the 'writable' verifier |
|||||
Issue descriptionCQ failure: https://luci-milo.appspot.com/buildbot/chromeos/guado_moblab-paladin/5514 Destroyed the moblab DUT. It is currently in a verify-repair loop. Provisioning the DUT failed with: touch: cannot touch '/var/tmp/provision_failed': Read-only file system Traceback (most recent call last): File "/usr/local/autotest/client/common_lib/test.py", line 817, in _call_test_function return func(*args, **dargs) File "/usr/local/autotest/client/common_lib/test.py", line 470, in execute dargs) File "/usr/local/autotest/client/common_lib/test.py", line 347, in _call_run_once_with_retry postprocess_profiled_run, args, dargs) File "/usr/local/autotest/client/common_lib/test.py", line 380, in _call_run_once self.run_once(*args, **dargs) File "/usr/local/autotest/server/site_tests/provision_AutoUpdate/provision_AutoUpdate.py", line 113, in run_once force_full_update=force) File "/usr/local/autotest/server/afe_utils.py", line 208, in machine_install_and_update_labels image_name, host_attributes = host.machine_install(*args, **dargs) File "/usr/local/autotest/server/hosts/cros_host.py", line 862, in machine_install self.run('touch %s' % self.PROVISION_FAILED) File "/usr/local/autotest/server/hosts/ssh_host.py", line 295, in run ssh_failure_retry_ok) File "/usr/local/autotest/server/hosts/ssh_host.py", line 252, in _run raise error.AutoservRunError("command execution error", result) Failed provision: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos2-row1-rack8-host1/421528-provision Following repair: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos2-row1-rack8-host1/421530-repair Following verify: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos2-row1-rack8-host1/421539-verify (it's in a verify-repair loop since) and the job _before_ the failed provision: http://cautotest/afe/#tab_id=view_job&object_id=110315076
,
Apr 13 2017
,
Apr 18 2017
,
Jul 24 2017
The Moblab instance failed because stateful was marked read-only, presumably because of some sort of transient FS failure. Verify for Moblab doesn't include 'writable', which checks for this condition. We should update the Moblab verify DAG to include this verifier, and the verifier should trigger the the 'reboot' repair action.
,
Jul 24 2017
This would make moblab failures more obvious because verify will fail instead of the test.
,
Jun 8 2018
Hi, this bug has not been updated recently. Please acknowledge the bug and provide status within two weeks (6/22/2018), or the bug will be archived. Thank you. |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by pprabhu@chromium.org
, Apr 3 2017