New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 821527 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Last visit > 30 days ago
Closed: Mar 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: ----



Sign in to add a comment

Test_push consistently fails in ssp setup due to "special device /host/etc/chrome-infra does not exist"

Project Member Reported by xixuan@chromium.org, Mar 13 2018

Issue description

After the emergency push, I did test_push, but it fails for the same reason twice:

Example failed job:
http://chromeos-staging-master2.hot.corp.google.com/afe/#tab_id=view_job&object_id=13477
http://chromeos-staging-master2.hot.corp.google.com/afe/#tab_id=view_job&object_id=13471

Logs (since redirect link is not working):
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/13477-chromeos-test/chromeos2-row1-rack2-host9
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/13471-chromeos-test/chromeos2-row1-rack2-host1

Traceback:
03/13 11:59:15.475 INFO |        server_job:0218| FAIL	----	----	timestamp=1520967555	localtime=Mar 13 11:59:15	Failed to setup container for test: Command <sudo lxc-attach -P /usr/local/autotest/containers -n container.OWfho3 -- bash -c "mount --bind /host/etc/chrome-infra /etc/chrome-infra"> failed, rc=32, Command returned non-zero exit status
  * Command: 
      sudo lxc-attach -P /usr/local/autotest/containers -n container.OWfho3 --
      bash -c "mount --bind /host/etc/chrome-infra /etc/chrome-infra"
  Exit status: 32
  Duration: 0.406045198441
  
  stderr:
  mount: special device /host/etc/chrome-infra does not exist. Check logs in ssp_logs folder for more details.
03/13 11:59:15.476 DEBUG|             utils:0214| Running 'sudo -n chown -R 99158 "/usr/local/autotest/results/13477-chromeos-test/chromeos2-row1-rack2-host9"'
03/13 11:59:15.885 DEBUG|             utils:0214| Running 'sudo -n chgrp -R 5000 "/usr/local/autotest/results/13477-chromeos-test/chromeos2-row1-rack2-host9"'
03/13 11:59:16.228 ERROR|         traceback:0013| Traceback (most recent call last):
03/13 11:59:16.229 ERROR|         traceback:0013|   File "/usr/local/autotest/server/autoserv", line 507, in run_autoserv
03/13 11:59:16.230 ERROR|         traceback:0013|     machines)
03/13 11:59:16.230 ERROR|         traceback:0013|   File "/usr/local/autotest/server/autoserv", line 168, in _run_with_ssp
03/13 11:59:16.230 ERROR|         traceback:0013|     dut_name=dut_name)
03/13 11:59:16.230 ERROR|         traceback:0013|   File "/usr/local/autotest/site-packages/chromite/lib/metrics.py", line 490, in wrapper
03/13 11:59:16.232 ERROR|         traceback:0013|     return fn(*args, **kwargs)
03/13 11:59:16.235 ERROR|         traceback:0013|   File "/usr/local/autotest/site_utils/lxc/cleanup_if_fail.py", line 40, in func_cleanup_if_fail
03/13 11:59:16.236 ERROR|         traceback:0013|     return func(*args, **kwargs)
03/13 11:59:16.236 ERROR|         traceback:0013|   File "/usr/local/autotest/site_utils/lxc/container_bucket.py", line 208, in setup_test
03/13 11:59:16.236 ERROR|         traceback:0013|     deploy_config_manager.deploy_pre_start()
03/13 11:59:16.237 ERROR|         traceback:0013|   File "/usr/local/autotest/site_utils/lxc/config.py", line 391, in deploy_pre_start
03/13 11:59:16.237 ERROR|         traceback:0013|     mount_config.readonly)
03/13 11:59:16.237 ERROR|         traceback:0013|   File "/usr/local/autotest/site_utils/lxc/zygote.py", line 214, in mount_dir
03/13 11:59:16.238 ERROR|         traceback:0013|     (container_host_path, destination))
03/13 11:59:16.238 ERROR|         traceback:0013|   File "/usr/local/autotest/site_utils/lxc/container.py", line 314, in attach_run
03/13 11:59:16.238 ERROR|         traceback:0013|     return utils.run(cmd)
03/13 11:59:16.239 ERROR|         traceback:0013|   File "/usr/local/autotest/client/common_lib/utils.py", line 748, in run
03/13 11:59:16.240 ERROR|         traceback:0013|     "Command returned non-zero exit status")
03/13 11:59:16.241 ERROR|         traceback:0013| CmdError: Command <sudo lxc-attach -P /usr/local/autotest/containers -n container.OWfho3 -- bash -c "mount --bind /host/etc/chrome-infra /etc/chrome-infra"> failed, rc=32, Command returned non-zero exit status
03/13 11:59:16.241 ERROR|         traceback:0013| * Command: 
03/13 11:59:16.241 ERROR|         traceback:0013|     sudo lxc-attach -P /usr/local/autotest/containers -n container.OWfho3 --
03/13 11:59:16.241 ERROR|         traceback:0013|     bash -c "mount --bind /host/etc/chrome-infra /etc/chrome-infra"
03/13 11:59:16.241 ERROR|         traceback:0013| Exit status: 32
03/13 11:59:16.241 ERROR|         traceback:0013| Duration: 0.406045198441
03/13 11:59:16.242 ERROR|         traceback:0013| 
03/13 11:59:16.242 ERROR|         traceback:0013| stderr:
03/13 11:59:16.242 ERROR|         traceback:0013| mount: special device /host/etc/chrome-infra does not exist
03/13 11:59:16.253 INFO |            client:0570| Attempting refresh to obtain initial access_token
03/13 11:59:16.323 INFO |            client:0872| Refreshing access_token
03/13 11:59:16.454 ERROR|          autoserv:0759| Uncaught SystemExit with code 1


+deputy in case prod also has the same failure.
 

Comment 1 by xixuan@chromium.org, Mar 13 2018

Labels: -Build

Comment 2 by xixuan@chromium.org, Mar 13 2018

Owner: jkop@chromium.org
Status: Assigned (was: Untriaged)

Comment 3 by jkop@chromium.org, Mar 14 2018

Status: Fixed (was: Assigned)
Revert fixed original instance, I'm working on relanding it without a recurrence.

Comment 4 by jkop@chromium.org, Mar 14 2018

Status: Tracking (was: Fixed)

Comment 5 by jkop@chromium.org, Mar 14 2018

Status: Fixed (was: Tracking)
Project Member

Comment 6 by bugdroid1@chromium.org, Mar 15 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/52a8d3f292a77ae9bed502274fd59eca24983d58

commit 52a8d3f292a77ae9bed502274fd59eca24983d58
Author: Jacob Kopczynski <jkop@google.com>
Date: Thu Mar 15 22:27:15 2018

autotest: lxc: factor out sudos in utils

Create a helper function to combine commands into a single `sudo bash -c` call.
This change is relanding a reverted attempt for the third time, with
still fewer moving parts and a feature flag.
Unlike previous attempts, this does not include any calls to the new
method.

BUG=chromium:809306
BUG= chromium:817569 
BUG= chromium:821527 
TEST=tryjob

Change-Id: Icbc1632ec9a9898fd234c7261421dda2ef10bc89
Reviewed-on: https://chromium-review.googlesource.com/963515
Commit-Ready: Jacob Kopczynski <jkop@chromium.org>
Tested-by: Jacob Kopczynski <jkop@chromium.org>
Reviewed-by: Jacob Kopczynski <jkop@chromium.org>

[modify] https://crrev.com/52a8d3f292a77ae9bed502274fd59eca24983d58/site_utils/lxc/utils.py
[modify] https://crrev.com/52a8d3f292a77ae9bed502274fd59eca24983d58/global_config.ini

Sign in to add a comment