New issue
Advanced search Search tips

Issue 707869 link

Starred by 2 users

Issue metadata

Status: Available
Owner: ----
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug

Blocking:
issue 747056



Sign in to add a comment

moblab verify should include the 'writable' verifier

Project Member Reported by pprabhu@chromium.org, Apr 3 2017

Issue description

CQ failure: https://luci-milo.appspot.com/buildbot/chromeos/guado_moblab-paladin/5514

Destroyed the moblab DUT. It is currently in a verify-repair loop.

Provisioning the DUT failed with:

touch: cannot touch '/var/tmp/provision_failed': Read-only file system
Traceback (most recent call last):
  File "/usr/local/autotest/client/common_lib/test.py", line 817, in _call_test_function
    return func(*args, **dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 470, in execute
    dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 347, in _call_run_once_with_retry
    postprocess_profiled_run, args, dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 380, in _call_run_once
    self.run_once(*args, **dargs)
  File "/usr/local/autotest/server/site_tests/provision_AutoUpdate/provision_AutoUpdate.py", line 113, in run_once
    force_full_update=force)
  File "/usr/local/autotest/server/afe_utils.py", line 208, in machine_install_and_update_labels
    image_name, host_attributes = host.machine_install(*args, **dargs)
  File "/usr/local/autotest/server/hosts/cros_host.py", line 862, in machine_install
    self.run('touch %s' % self.PROVISION_FAILED)
  File "/usr/local/autotest/server/hosts/ssh_host.py", line 295, in run
    ssh_failure_retry_ok)
  File "/usr/local/autotest/server/hosts/ssh_host.py", line 252, in _run
    raise error.AutoservRunError("command execution error", result)


Failed provision: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos2-row1-rack8-host1/421528-provision
Following repair: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos2-row1-rack8-host1/421530-repair
Following verify: https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/hosts/chromeos2-row1-rack8-host1/421539-verify
(it's in a verify-repair loop since)

and the job _before_ the failed provision: http://cautotest/afe/#tab_id=view_job&object_id=110315076
 
We're talking about chromeos2-row1-rack8-host1 here.

Filed b/36860221 to reclaim this DUT.

Comment 2 by aut...@google.com, Apr 13 2017

Owner: jrbarnette@chromium.org

Comment 3 by aut...@google.com, Apr 18 2017

Labels: -current-issue
Labels: Hotlist-Fixit
Owner: ----
Status: Available (was: Untriaged)
Summary: moblab verify should include the 'writable' verifier (was: moblab DUT stuck failing provision with "touch: cannot touch '/var/tmp/provision_failed'")
The Moblab instance failed because stateful was marked read-only,
presumably because of some sort of transient FS failure.

Verify for Moblab doesn't include 'writable', which checks for
this condition.  We should update the Moblab verify DAG to
include this verifier, and the verifier should trigger the
the 'reboot' repair action.

Blocking: 747056
This would make moblab failures more obvious because verify will fail instead of the test.
Hi, this bug has not been updated recently. Please acknowledge the bug and provide status within two weeks (6/22/2018), or the bug will be archived. Thank you.

Sign in to add a comment