Chrome Version: R69-10895.21.0
OS: Chrome OS
DUT: Nautilus
Hostname: chromeos1-row2-rack7-host3
What steps will reproduce the problem?
(1) DUT is in "Repari Failed" state.
(2) ssh to dut to check for python. No python available.
(3) run "dev_install --reinstall". dev_install completed successfully.
(4) Trigger a "Reverify" job. Reverify job fails and runs "repair" job.
What is the expected result?
Repair job to be successful.
What happens instead?
Repair fails as python goes missing again.
Logs@
https://stainless.corp.google.com/browse/chromeos-autotest-results/hosts/chromeos1-row2-rack7-host3/64482662-repair/
snippet below:
08/22 14:18:00.358 DEBUG| dev_server:0790| Pick http://172.27.215.246:8082
08/22 14:18:00.358 INFO | dev_server:1255| Staging artifacts on devserver http://172.27.215.246:8082: build=nautilus-release/R69-10895.21.0, artifacts=['test_image'], files=, archive_url=gs://chromeos-image-archive/nautilus-release/R69-10895.21.0
08/22 14:18:00.359 DEBUG| utils:0218| Running 'ssh 172.27.215.246 'curl "http://172.27.215.246:8082/stage?artifacts=test_image&files=&async=True&archive_url=gs://chromeos-image-archive/nautilus-release/R69-10895.21.0"''
08/22 14:18:00.689 DEBUG| dev_server:1197| response for RPC: '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"\n"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">\n<html>\n<head>\n <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>\n <title>500 Internal Server Error</title>\n <style type="text/css">\n #powered_by {\n margin-top: 20px;\n border-top: 2px solid black;\n font-style: italic;\n }\n\n #traceback {\n color: red;\n }\n </style>\n</head>\n <body>\n <h2>500 Internal Server Error</h2>\n <p>The server encountered an unexpected condition which prevented it from fulfilling the request.</p>\n <pre id="traceback">Traceback (most recent call last):\n File "/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 656, in respond\n response.body = self.handler()\n File "/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line 188, in __call__\n self.body = self.oldhandler(*args, **kwargs)\n File "/usr/lib/python2.7/dist-packages/cherrypy/_cpdispatch.py", line 34, in __call__\n return self.callable(*self.args, **self.kwargs)\n File "/home/chromeos-test/chromiumos/src/platform/dev/devserver.py", line 891, in stage\n dl.Download(factory, async=async)\n File "/home/chromeos-test/chromiumos/src/platform/dev/downloader.py", line 178, in Download\n common_util.MkDirP(self._build_dir)\n File "/home/chromeos-test/chromiumos/src/platform/dev/common_util.py", line 60, in MkDirP\n os.makedirs(directory)\n File "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name, mode)\nOSError: [Errno 30] Read-only file system: \'/home/chromeos-test/images/nautilus-release/R69-10895.21.0\'\n</pre>\n <div id="powered_by">\n <span>Powered by <a href="http://www.cherrypy.org">CherryPy 3.2.2</a></span>\n </div>\n </body>\n</html>\n'
08/22 14:18:00.692 ERROR| repair:0507| Repair failed: Reinstall from USB using servo
Traceback (most recent call last):
File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 505, in _repair_host
self.repair(host)
File "/usr/local/autotest/server/hosts/cros_repair.py", line 566, in repair
host.servo_install(host.stage_image_for_servo())
File "/usr/local/autotest/server/hosts/cros_host.py", line 484, in stage_image_for_servo
devserver.stage_artifacts(image_name, [artifact])
File "/usr/local/autotest/client/common_lib/cros/dev_server.py", line 376, in metrics_wrapper
return wrapper()
File "/usr/local/autotest/client/common_lib/cros/retry.py", line 218, in func_retry
remaining_time)
File "/usr/local/autotest/client/common_lib/cros/retry.py", line 123, in timeout
default_result = func(*args, **kwargs)
File "/usr/local/autotest/client/common_lib/cros/dev_server.py", line 370, in wrapper
return method(*args, **kwargs)
File "/usr/local/autotest/client/common_lib/cros/dev_server.py", line 1578, in stage_artifacts
self._stage_artifacts(image, artifacts, files, archive_url)
File "/usr/local/autotest/client/common_lib/cros/dev_server.py", line 1272, in _stage_artifacts
**arguments)
File "/usr/local/autotest/client/common_lib/cros/dev_server.py", line 376, in metrics_wrapper
return wrapper()
File "/usr/local/autotest/client/common_lib/cros/retry.py", line 218, in func_retry
remaining_time)
File "/usr/local/autotest/client/common_lib/cros/retry.py", line 123, in timeout
default_result = func(*args, **kwargs)
File "/usr/local/autotest/client/common_lib/cros/dev_server.py", line 370, in wrapper
return method(*args, **kwargs)
File "/usr/local/autotest/client/common_lib/cros/dev_server.py", line 1551, in call_and_wait
expected_response, **kwargs)
File "/usr/local/autotest/client/common_lib/cros/dev_server.py", line 1210, in _call_and_wait
raise DevServerException(error_message)
DevServerException: staging artifacts=test_image files= for nautilus-release/R69-10895.21.0 failed;HTTP OK not accompanied by 'Success'.
08/22 14:18:00.693 INFO | server_job:0216| FAIL ---- repair.usb timestamp=1534972680 localtime=Aug 22 14:18:00 staging artifacts=test_image files= for nautilus-release/R69-10895.21.0 failed;HTTP OK not accompanied by 'Success'.
08/22 14:18:00.694 INFO | server_job:0216| END FAIL ---- repair.usb timestamp=1534972680 localtime=Aug 22 14:18:00
08/22 14:18:00.694 INFO | repair:0349| Verifying this condition: The host should have valid HWID and Serial Number
08/22 14:18:00.701 DEBUG| ssh_host:0301| Running (ssh) 'crossystem hwid' from '_verify_dependencies|_verify_list|_verify_host|verify|run|run_very_slowly'
Comment 1 by dsunk...@chromium.org
, Aug 2208/22 14:18:02.615 DEBUG| repair:0113| Did not find critical file system errors 08/22 14:18:02.615 DEBUG| repair:0113| The firmware on this DUT is up-to-date 08/22 14:18:02.616 ERROR| repair:0048| Repair failed due to Exception. Traceback (most recent call last): File "/usr/local/autotest/server/control_segments/repair", line 38, in repair target.repair() File "/usr/local/autotest/server/hosts/cros_host.py", line 740, in repair self._repair_strategy.repair(self) File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 774, in repair self._verify_root._verify_host(host, silent) File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 348, in _verify_host self._verify_dependencies(host, silent) File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 226, in _verify_dependencies self._verify_list(host, self._dependency_list, silent) File "/usr/local/autotest/client/common_lib/hosts/repair.py", line 216, in _verify_list raise AutoservVerifyDependencyError(self, failures) AutoservVerifyDependencyError: Python is missing; may be caused by powerwash Saw file system error: [ 1.983786] EXT4-fs error (device mmcblk0p1): ext4_has_uninit_itable:2883: comm mount: Inode table for bg 0 marked as needing zeroing DUT firmware requires update from Google_Nautilus.10431.58.0 to Google_Nautilus.10431.67.0 08/22 14:18:02.616 INFO | server_job:0216| END FAIL ---- repair timestamp=1534972682 localtime=Aug 22 14:18:02 08/22 14:18:02.690 DEBUG| servo:0155| Stop capturing CPU/EC UART. 08/22 14:18:02.690 INFO | servo:0588| Setting cpu_uart_capture to 'off' 08/22 14:18:02.834 INFO | servo:0588| Setting ec_uart_capture to 'off'