AU Failed during provision for stumpy_paladin. Not sure if this is flake or not, seems the failure occured in the stateful_update script.


https://uberchromegw.corp.google.com/i/chromeos/builders/stumpy%20paladin/builds/24228


https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/55456237-chromeos-test/chromeos2-row3-rack4-host1/debug/?debugUI=CLOUD


02/29 09:39:48.313 DEBUG|          ssh_host:0153| Running (ssh) '/tmp/stateful_update --stateful_change=reset 2>&1'
02/29 09:39:48.519 DEBUG|        base_utils:0268| [stdout] Resetting stateful update state.
02/29 09:39:52.738 DEBUG|        base_utils:0268| [stdout] rm: cannot remove '/mnt/stateful_partition/dev_image_new': Directory not empty
02/29 09:39:52.741 WARNI|         cros_host:0770| Autoupdate did not complete.
02/29 09:39:52.752 WARNI|              test:0543| Autotest caught exception when running test:
Traceback (most recent call last):
  File "/usr/local/autotest/client/common_lib/test.py", line 537, in _exec
    _call_test_function(self.execute, *p_args, **p_dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 747, in _call_test_function
    raise error.UnhandledTestFail(e)
UnhandledTestFail: Unhandled AutoservRunError: command execution error
* Command: 
    /usr/bin/ssh -a -x    -o ControlPath=/tmp/_autotmp_OaamXDssh-
    master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
    -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o
    ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22
    chromeos2-row3-rack4-host1 "export LIBC_FATAL_STDERR_=1;
    /tmp/stateful_update --stateful_change=reset 2>&1"
Exit status: 1
Duration: 4.38842391968

stdout:
Resetting stateful update state.
rm: cannot remove '/mnt/stateful_partition/dev_image_new': Directory not empty
Traceback (most recent call last):
  File "/usr/local/autotest/client/common_lib/test.py", line 741, in _call_test_function
    return func(*args, **dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 398, in execute
    dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 284, in _call_run_once_with_retry
    postprocess_profiled_run, args, dargs)
  File "/usr/local/autotest/client/common_lib/test.py", line 313, in _call_run_once
    self.run_once(*args, **dargs)
  File "/usr/local/autotest/server/site_tests/provision_AutoUpdate/provision_AutoUpdate.py", line 136, in run_once
    force_full_update=force)
  File "/usr/local/autotest/server/afe_utils.py", line 192, in machine_install_and_update_labels
    image_name, host_attributes = host.machine_install(*args, **dargs)
  File "/usr/local/autotest/server/hosts/cros_host.py", line 768, in machine_install
    updater.run_update()
  File "/usr/local/autotest/site-packages/statsd/timer.py", line 95, in _decorator
    return function(*args, **kwargs)
  File "/usr/local/autotest/client/common_lib/cros/autoupdater.py", line 474, in run_update
    self.reset_stateful_partition()
  File "/usr/local/autotest/client/common_lib/cros/autoupdater.py", line 360, in reset_stateful_partition
    self._run(' '.join(statefuldev_cmd))
  File "/usr/local/autotest/client/common_lib/cros/autoupdater.py", line 268, in _run
    return self.host.run(cmd, *args, **kwargs)
  File "/usr/local/autotest/server/hosts/ssh_host.py", line 162, in run
    options, stdin, args, ignore_timeout)
  File "/usr/local/autotest/server/hosts/ssh_host.py", line 130, in _run
    raise error.AutoservRunError("command execution error", result)
AutoservRunError: command execution error
* Command: 
    /usr/bin/ssh -a -x    -o ControlPath=/tmp/_autotmp_OaamXDssh-
    master/socket -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
    -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o
    ServerAliveCountMax=3 -o ConnectionAttempts=4 -o Protocol=2 -l root -p 22
    chromeos2-row3-rack4-host1 "export LIBC_FATAL_STDERR_=1;
    /tmp/stateful_update --stateful_change=reset 2>&1"
Exit status: 1
Duration: 4.38842391968

stdout:
Resetting stateful update state.
rm: cannot remove '/mnt/stateful_partition/dev_image_new': Directory not empty