Test_push consistently fails in ssp setup due to "special device /host/etc/chrome-infra does not exist" |
|||||
Issue descriptionAfter the emergency push, I did test_push, but it fails for the same reason twice: Example failed job: http://chromeos-staging-master2.hot.corp.google.com/afe/#tab_id=view_job&object_id=13477 http://chromeos-staging-master2.hot.corp.google.com/afe/#tab_id=view_job&object_id=13471 Logs (since redirect link is not working): https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/13477-chromeos-test/chromeos2-row1-rack2-host9 https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/13471-chromeos-test/chromeos2-row1-rack2-host1 Traceback: 03/13 11:59:15.475 INFO | server_job:0218| FAIL ---- ---- timestamp=1520967555 localtime=Mar 13 11:59:15 Failed to setup container for test: Command <sudo lxc-attach -P /usr/local/autotest/containers -n container.OWfho3 -- bash -c "mount --bind /host/etc/chrome-infra /etc/chrome-infra"> failed, rc=32, Command returned non-zero exit status * Command: sudo lxc-attach -P /usr/local/autotest/containers -n container.OWfho3 -- bash -c "mount --bind /host/etc/chrome-infra /etc/chrome-infra" Exit status: 32 Duration: 0.406045198441 stderr: mount: special device /host/etc/chrome-infra does not exist. Check logs in ssp_logs folder for more details. 03/13 11:59:15.476 DEBUG| utils:0214| Running 'sudo -n chown -R 99158 "/usr/local/autotest/results/13477-chromeos-test/chromeos2-row1-rack2-host9"' 03/13 11:59:15.885 DEBUG| utils:0214| Running 'sudo -n chgrp -R 5000 "/usr/local/autotest/results/13477-chromeos-test/chromeos2-row1-rack2-host9"' 03/13 11:59:16.228 ERROR| traceback:0013| Traceback (most recent call last): 03/13 11:59:16.229 ERROR| traceback:0013| File "/usr/local/autotest/server/autoserv", line 507, in run_autoserv 03/13 11:59:16.230 ERROR| traceback:0013| machines) 03/13 11:59:16.230 ERROR| traceback:0013| File "/usr/local/autotest/server/autoserv", line 168, in _run_with_ssp 03/13 11:59:16.230 ERROR| traceback:0013| dut_name=dut_name) 03/13 11:59:16.230 ERROR| traceback:0013| File "/usr/local/autotest/site-packages/chromite/lib/metrics.py", line 490, in wrapper 03/13 11:59:16.232 ERROR| traceback:0013| return fn(*args, **kwargs) 03/13 11:59:16.235 ERROR| traceback:0013| File "/usr/local/autotest/site_utils/lxc/cleanup_if_fail.py", line 40, in func_cleanup_if_fail 03/13 11:59:16.236 ERROR| traceback:0013| return func(*args, **kwargs) 03/13 11:59:16.236 ERROR| traceback:0013| File "/usr/local/autotest/site_utils/lxc/container_bucket.py", line 208, in setup_test 03/13 11:59:16.236 ERROR| traceback:0013| deploy_config_manager.deploy_pre_start() 03/13 11:59:16.237 ERROR| traceback:0013| File "/usr/local/autotest/site_utils/lxc/config.py", line 391, in deploy_pre_start 03/13 11:59:16.237 ERROR| traceback:0013| mount_config.readonly) 03/13 11:59:16.237 ERROR| traceback:0013| File "/usr/local/autotest/site_utils/lxc/zygote.py", line 214, in mount_dir 03/13 11:59:16.238 ERROR| traceback:0013| (container_host_path, destination)) 03/13 11:59:16.238 ERROR| traceback:0013| File "/usr/local/autotest/site_utils/lxc/container.py", line 314, in attach_run 03/13 11:59:16.238 ERROR| traceback:0013| return utils.run(cmd) 03/13 11:59:16.239 ERROR| traceback:0013| File "/usr/local/autotest/client/common_lib/utils.py", line 748, in run 03/13 11:59:16.240 ERROR| traceback:0013| "Command returned non-zero exit status") 03/13 11:59:16.241 ERROR| traceback:0013| CmdError: Command <sudo lxc-attach -P /usr/local/autotest/containers -n container.OWfho3 -- bash -c "mount --bind /host/etc/chrome-infra /etc/chrome-infra"> failed, rc=32, Command returned non-zero exit status 03/13 11:59:16.241 ERROR| traceback:0013| * Command: 03/13 11:59:16.241 ERROR| traceback:0013| sudo lxc-attach -P /usr/local/autotest/containers -n container.OWfho3 -- 03/13 11:59:16.241 ERROR| traceback:0013| bash -c "mount --bind /host/etc/chrome-infra /etc/chrome-infra" 03/13 11:59:16.241 ERROR| traceback:0013| Exit status: 32 03/13 11:59:16.241 ERROR| traceback:0013| Duration: 0.406045198441 03/13 11:59:16.242 ERROR| traceback:0013| 03/13 11:59:16.242 ERROR| traceback:0013| stderr: 03/13 11:59:16.242 ERROR| traceback:0013| mount: special device /host/etc/chrome-infra does not exist 03/13 11:59:16.253 INFO | client:0570| Attempting refresh to obtain initial access_token 03/13 11:59:16.323 INFO | client:0872| Refreshing access_token 03/13 11:59:16.454 ERROR| autoserv:0759| Uncaught SystemExit with code 1 +deputy in case prod also has the same failure.
,
Mar 13 2018
,
Mar 14 2018
Revert fixed original instance, I'm working on relanding it without a recurrence.
,
Mar 14 2018
,
Mar 14 2018
,
Mar 15 2018
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/52a8d3f292a77ae9bed502274fd59eca24983d58 commit 52a8d3f292a77ae9bed502274fd59eca24983d58 Author: Jacob Kopczynski <jkop@google.com> Date: Thu Mar 15 22:27:15 2018 autotest: lxc: factor out sudos in utils Create a helper function to combine commands into a single `sudo bash -c` call. This change is relanding a reverted attempt for the third time, with still fewer moving parts and a feature flag. Unlike previous attempts, this does not include any calls to the new method. BUG=chromium:809306 BUG= chromium:817569 BUG= chromium:821527 TEST=tryjob Change-Id: Icbc1632ec9a9898fd234c7261421dda2ef10bc89 Reviewed-on: https://chromium-review.googlesource.com/963515 Commit-Ready: Jacob Kopczynski <jkop@chromium.org> Tested-by: Jacob Kopczynski <jkop@chromium.org> Reviewed-by: Jacob Kopczynski <jkop@chromium.org> [modify] https://crrev.com/52a8d3f292a77ae9bed502274fd59eca24983d58/site_utils/lxc/utils.py [modify] https://crrev.com/52a8d3f292a77ae9bed502274fd59eca24983d58/global_config.ini |
|||||
►
Sign in to add a comment |
|||||
Comment 1 by xixuan@chromium.org
, Mar 13 2018