TradefedTest: incomplete cleanup |
||||
Issue description
An exception may happen in the test's cleanup() that prevents subsequent cleanup steps from running. In particular, if _run_adb_cmd() throws an exception then shutil.rmtree() doesn't get called.
Example:
> 09/26 14:26:08.685 DEBUG| test:0611| Running cleanup for test.
> 09/26 14:26:08.686 DEBUG| utils:0219| Running 'adb -s 'chromeos6-row1-rack24-host13:22' kill-server'
> 09/26 14:26:08.753 ERROR| test:0614| Ignoring exception during cleanup() phase:
> 09/26 14:26:08.753 ERROR| traceback:0013| Traceback (most recent call last):
> 09/26 14:26:08.754 ERROR| traceback:0013| File "/usr/local/autotest/client/common_lib/test.py", line 612, in _exec
> 09/26 14:26:08.754 ERROR| traceback:0013| _cherry_pick_call(self.cleanup, *args, **dargs)
> 09/26 14:26:08.754 ERROR| traceback:0013| File "/usr/local/autotest/client/common_lib/test.py", line 715, in _cherry_pick_call
> 09/26 14:26:08.755 ERROR| traceback:0013| return func(*p_args, **p_dargs)
> 09/26 14:26:08.755 ERROR| traceback:0013| File "/usr/local/autotest/server/cros/tradefed_test.py", line 154, in cleanup
> 09/26 14:26:08.756 ERROR| traceback:0013| self._run_adb_cmd(host, verbose=True, args=('kill-server',))
> 09/26 14:26:08.756 ERROR| traceback:0013| File "/usr/local/autotest/server/cros/tradefed_test.py", line 262, in _run_adb_cmd
> 09/26 14:26:08.756 ERROR| traceback:0013| result = self._run('adb', **kwargs)
> 09/26 14:26:08.757 ERROR| traceback:0013| File "/usr/local/autotest/server/cros/tradefed_test.py", line 709, in _run
> 09/26 14:26:08.757 ERROR| traceback:0013| return utils.run(*args, **kwargs)
> 09/26 14:26:08.757 ERROR| traceback:0013| File "/usr/local/autotest/client/common_lib/utils.py", line 759, in run
> 09/26 14:26:08.783 ERROR| traceback:0013| "Command returned non-zero exit status")
> 09/26 14:26:08.788 ERROR| traceback:0013| CmdError: Command <adb -s 'chromeos6-row1-rack24-host13:22' kill-server> failed, rc=127, Command returned non-zero exit status
> 09/26 14:26:08.789 ERROR| traceback:0013| * Command:
> 09/26 14:26:08.789 ERROR| traceback:0013| adb -s 'chromeos6-row1-rack24-host13:22' kill-server
> 09/26 14:26:08.790 ERROR| traceback:0013| Exit status: 127
> 09/26 14:26:08.790 ERROR| traceback:0013| Duration: 0.0207178592682
> 09/26 14:26:08.790 ERROR| traceback:0013|
> 09/26 14:26:08.790 ERROR| traceback:0013| stderr:
> 09/26 14:26:08.791 ERROR| traceback:0013| /bin/bash: adb: command not found
https://luci-milo.appspot.com/buildbot/chromeos/kevin-arcnext-paladin/1448
,
Sep 28
This is harmless. When the cache lock fails it doesn't have adb binaries, so nothing to do here. And the cleanup in the lab is ultimately done by destroying the lxc container. So the secondary error is harmless in this case. I guess on a local machine it would be more problematic. 09/26 14:26:08.636 ERROR| tradefed_utils:0047| Permanent lock failure. Trying to break lock. Kazuhiro, do you think we should catch errors in tradefed_test.py cleanup()?
,
Sep 28
I'm not too much worried about local machine runs, but for the lab runs, that change will be nice for sheriffs/constables, I think. People tends to focus on the last failing stack trace first, but in this case that one is confusing. The thing that really needs to be emphasized is an early setUp() phase failure that blocked the test getting adb binary.
,
Oct 10
,
Oct 11
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/bd5391adf559d5a93585e4e9876cb5697c3ee194 commit bd5391adf559d5a93585e4e9876cb5697c3ee194 Author: Ilja H. Friedel <ihf@chromium.org> Date: Thu Oct 11 20:25:57 2018 tradefed_test: clean up cleanup(). Move up initialization of path so that a failure to ssh into the DUT does not cause accessing it not initialized (crbug.com/893455) Catch exceptions to suppress possible problems during cleanup. BUG= chromium:889771 , chromium:893455 TEST=None. Change-Id: If66cdc2bacf35796e6547de499917dcb463a730a Reviewed-on: https://chromium-review.googlesource.com/1271927 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Ilja H. Friedel <ihf@chromium.org> Reviewed-by: Pohsien Wang <pwang@chromium.org> Reviewed-by: Keith Haddow <haddowk@chromium.org> [modify] https://crrev.com/bd5391adf559d5a93585e4e9876cb5697c3ee194/server/cros/tradefed_test.py
,
Nov 14
|
||||
►
Sign in to add a comment |
||||
Comment 1 by emaxx@chromium.org
, Sep 27