New issue
Advanced search Search tips

Issue 922160 link

Starred by 2 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

provision failure on grunt-chrome-pfq

Project Member Reported by afakhry@chromium.org, Jan 15

Issue description

- DUT: chromeos2-row3-rack9-host19

- grunt-chrome-pfq: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8924262175300252176

@@@STEP_LINK@[Test-Logs]: provision: FAIL: Download and install failed from chromeos4-devserver8.cros.corp.google.com onto chromeos2-row3-rack9-host19: ('ssh timed out', * Command: @http://cautotest-prod/tko/retrieve_logs.cgi?job=/results/277222957-chromeos-test/@@@
    Will return from run_suite with status: INFRA_FAILURE

stderr:
ssh: connect to host chromeos2-row3-rack9-host19 port 22: Connection timed out)
01/15 10:03:18.709 DEBUG|              test:0381| Test failed due to Download and install failed from chromeos4-devserver8.cros.corp.google.com onto chromeos2-row3-rack9-host19: ('ssh timed out', * Command: 
    /usr/bin/ssh -a -x  -o ControlPath=/tmp/_autotmp_CTh7YKssh-master/socket
    -o Protocol=2 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
    -o BatchMode=yes -o ConnectTimeout=30 -o ServerAliveInterval=900 -o
    ServerAliveCountMax=3 -o ConnectionAttempts=4 -l root -p 22
    chromeos2-row3-rack9-host19 "export LIBC_FATAL_STDERR_=1; if type
    \"logger\" > /dev/null 2>&1; then logger -tag \"autotest\"
    \"server[stack::_rootdev|_run|run] -> ssh_run(rootdev -s)\";fi; rootdev
    -s"
Exit status: 255
Duration: 63.5128240585
 

Comment 1 by akes...@chromium.org, Jan 18 (5 days ago)


dut-status seems to indicate that this device is in a repair fail loop; locking it

$ dut-status -f chromeos2-row3-rack9-host19
chromeos2-row3-rack9-host19
    2019-01-17 15:21:59  -- https://stainless.corp.google.com/browse/chromeos-autotest-results/277958145-chromeos-test/
    2019-01-17 15:15:17  OK https://stainless.corp.google.com/browse/chromeos-autotest-results/hosts/chromeos2-row3-rack9-host19/1753297-provision/
    2019-01-17 12:40:39  OK https://stainless.corp.google.com/browse/chromeos-autotest-results/hosts/chromeos2-row3-rack9-host19/1751989-repair/
    2019-01-17 11:29:24  -- https://stainless.corp.google.com/browse/chromeos-autotest-results/hosts/chromeos2-row3-rack9-host19/1751293-provision/
    2019-01-17 10:45:20  OK https://stainless.corp.google.com/browse/chromeos-autotest-results/hosts/chromeos2-row3-rack9-host19/1750943-repair/
    2019-01-17 09:29:49  -- https://stainless.corp.google.com/browse/chromeos-autotest-results/hosts/chromeos2-row3-rack9-host19/1750218-provision/
    2019-01-17 04:52:01  -- https://stainless.corp.google.com/browse/chromeos-autotest-results/277835059-chromeos-test/
    2019-01-17 04:23:03  OK https://stainless.corp.google.com/browse/chromeos-autotest-results/hosts/chromeos2-row3-rack9-host19/1746757-provision/
    2019-01-17 03:07:12  -- https://stainless.corp.google.com/browse/chromeos-autotest-results/277804377-chromeos-test/
    2019-01-17 03:00:28  OK https://stainless.corp.google.com/browse/chromeos-autotest-results/hosts/chromeos2-row3-rack9-host19/1745783-provision/
    2019-01-16 23:26:38  -- https://stainless.corp.google.com/browse/chromeos-autotest-results/277670678-chromeos-test/
    2019-01-16 23:19:37  OK https://stainless.corp.google.com/browse/chromeos-autotest-results/hosts/chromeos2-row3-rack9-host19/1742894-provision/
    2019-01-16 21:51:58  -- https://stainless.corp.google.com/browse/chromeos-autotest-results/277648192-chromeos-test/
    2019-01-16 21:45:15  OK https://stainless.corp.google.com/browse/chromeos-autotest-results/hosts/chromeos2-row3-rack9-host19/1742430-provision/
    2019-01-16 19:03:01  -- https://stainless.corp.google.com/browse/chromeos-autotest-results/277637127-chromeos-test/
    2019-01-16 19:02:16  OK https://stainless.corp.google.com/browse/chromeos-autotest-results/hosts/chromeos2-row3-rack9-host19/1741166-reset/
    2019-01-16 18:48:06  -- https://stainless.corp.google.com/browse/chromeos-autotest-results/277636766-chromeos-test/
    2019-01-16 18:27:45  OK https://stainless.corp.google.com/browse/chromeos-autotest-results/hosts/chromeos2-row3-rack9-host19/1740802-provision/

Comment 2 by akes...@chromium.org, Jan 18 (5 days ago)

Labels: -Pri-1 Pri-2
shouldn't kill canary or pfq anymore, but DUT still needs to be revived eventually

Sign in to add a comment