New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 788437 link

Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 788455
Owner:
Closed: Nov 2017
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug

Blocked on:
issue 788455



Sign in to add a comment

reef-paladin is flaky

Project Member Reported by pgeorgi@chromium.org, Nov 24 2017

Issue description

Eg in https://luci-milo.appspot.com/buildbot/chromeos/reef-paladin/4358

It fails in provision_AutoUpdate.double_SERVER_JOB on basking and electro.
See also https://viceroy.corp.google.com/chromeos/suite_details?job_id=158313761
 
Labels: -Pri-3 Pri-1
Since it happens on tons of devices, it's probably not a hardware issue.

Log output (autoserv.INFO):

11/24 04:17:28.927 INFO |          autoserv:0685| Results placed in /usr/local/autotest/results/158314027-chromeos-test/chromeos6-row4-rack10-host3
11/24 04:17:28.928 INFO |           pidfile:0016| Logged pid 27953 to /usr/local/autotest/results/158314027-chromeos-test/chromeos6-row4-rack10-host3/.autoserv_execute
11/24 04:17:29.230 INFO |        server_job:1580| Shadowing AFE store with a FileStore at /usr/local/autotest/results/158314027-chromeos-test/chromeos6-row4-rack10-host3/host_info_store/store_a33a8d69-9a5c-4366-8fee-b6ab05d79f3f
11/24 04:17:29.351 INFO |    connectionpool:0207| Starting new HTTP connection (1): metadata.google.internal
11/24 04:17:29.670 NOTIC|      cros_logging:0038| ts_mon was set up.
11/24 04:19:42.784 INFO |        server_job:0213| FAIL	----	----	timestamp=1511525982	localtime=Nov 24 04:19:42	Failed to setup container for test: Command <sudo lxc-start -P /usr/local/autotest/containers -n test_158314027_1511525848_27953 -d> failed, rc=1, Command returned non-zero exit status
  * Command: 
      sudo lxc-start -P /usr/local/autotest/containers -n
      test_158314027_1511525848_27953 -d
  Exit status: 1
  Duration: 6.00990080833
  
  stderr:
  lxc-start: tools/lxc_start.c: main: 366 The container failed to start.
  lxc-start: tools/lxc_start.c: main: 368 To get more details, run the container in foreground mode.
  lxc-start: tools/lxc_start.c: main: 370 Additional information can be obtained by setting the --logfile and --logpriority options.. Check logs in ssp_logs folder for more details.
11/24 04:19:44.098 ERROR|         traceback:0013| Traceback (most recent call last):
11/24 04:19:44.099 ERROR|         traceback:0013|   File "/usr/local/autotest/server/autoserv", line 507, in run_autoserv
11/24 04:19:44.100 ERROR|         traceback:0013|     machines)
11/24 04:19:44.100 ERROR|         traceback:0013|   File "/usr/local/autotest/server/autoserv", line 168, in _run_with_ssp
11/24 04:19:44.101 ERROR|         traceback:0013|     dut_name=dut_name)
11/24 04:19:44.101 ERROR|         traceback:0013|   File "/usr/local/autotest/site-packages/chromite/lib/metrics.py", line 483, in wrapper
11/24 04:19:44.102 ERROR|         traceback:0013|     return fn(*args, **kwargs)
11/24 04:19:44.103 ERROR|         traceback:0013|   File "/usr/local/autotest/site_utils/lxc/cleanup_if_fail.py", line 40, in func_cleanup_if_fail
11/24 04:19:44.103 ERROR|         traceback:0013|     return func(*args, **kwargs)
11/24 04:19:44.104 ERROR|         traceback:0013|   File "/usr/local/autotest/site_utils/lxc/container_bucket.py", line 186, in setup_test
11/24 04:19:44.104 ERROR|         traceback:0013|     container.start(wait_for_network=True)
11/24 04:19:44.105 ERROR|         traceback:0013|   File "/usr/local/autotest/site-packages/chromite/lib/metrics.py", line 483, in wrapper
11/24 04:19:44.105 ERROR|         traceback:0013|     return fn(*args, **kwargs)
11/24 04:19:44.106 ERROR|         traceback:0013|   File "/usr/local/autotest/site_utils/lxc/container.py", line 319, in start
11/24 04:19:44.106 ERROR|         traceback:0013|     output = utils.run(cmd).stdout
11/24 04:19:44.107 ERROR|         traceback:0013|   File "/usr/local/autotest/client/common_lib/utils.py", line 738, in run
11/24 04:19:44.108 ERROR|         traceback:0013|     "Command returned non-zero exit status")
11/24 04:19:44.109 ERROR|         traceback:0013| CmdError: Command <sudo lxc-start -P /usr/local/autotest/containers -n test_158314027_1511525848_27953 -d> failed, rc=1, Command returned non-zero exit status
11/24 04:19:44.110 ERROR|         traceback:0013| * Command: 
11/24 04:19:44.110 ERROR|         traceback:0013|     sudo lxc-start -P /usr/local/autotest/containers -n
11/24 04:19:44.111 ERROR|         traceback:0013|     test_158314027_1511525848_27953 -d
11/24 04:19:44.111 ERROR|         traceback:0013| Exit status: 1
11/24 04:19:44.112 ERROR|         traceback:0013| Duration: 6.00990080833
11/24 04:19:44.112 ERROR|         traceback:0013| 
11/24 04:19:44.112 ERROR|         traceback:0013| stderr:
11/24 04:19:44.113 ERROR|         traceback:0013| lxc-start: tools/lxc_start.c: main: 366 The container failed to start.
11/24 04:19:44.113 ERROR|         traceback:0013| lxc-start: tools/lxc_start.c: main: 368 To get more details, run the container in foreground mode.
11/24 04:19:44.113 ERROR|         traceback:0013| lxc-start: tools/lxc_start.c: main: 370 Additional information can be obtained by setting the --logfile and --logpriority options.
11/24 04:19:44.136 INFO |            client:0570| Attempting refresh to obtain initial access_token
11/24 04:19:44.234 INFO |            client:0872| Refreshing access_token
11/24 04:19:44.677 ERROR|          autoserv:0759| Uncaught SystemExit with code 1
Traceback (most recent call last):
  File "/usr/local/autotest/server/autoserv", line 755, in main
    use_ssp)
  File "/usr/local/autotest/server/autoserv", line 562, in run_autoserv
    sys.exit(exit_code)
SystemExit: 1
Cc: cernekee@chromium.org
Owner: jeffcarp@chromium.org
Blockedon: 788455
Weird, my blocked CLs passed the CQ last night...
Labels: Infra-Troopers
Owner: ----
I am no longer trooper, reassigning to trooper queue.
Owner: akes...@chromium.org
Status: Assigned (was: Untriaged)
chromeos, assigning to akeshet for triage.
Mergedinto: 788455
Status: Duplicate (was: Assigned)

Sign in to add a comment