New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 665286 link

Starred by 1 user
Status: Duplicate
Owner: ----
Closed: Nov 2016
Cc:
█ ntang@chromium.org
█ pgeorgi@chromium.org
█ jrbarnette@chromium.org
skau@chromium.org
semenzato@chromium.org
pprabhu@chromium.org
█ sbasi@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

guado_moblab-paladin HWTest failing

Project Member Reported by skau@chromium.org, Nov 15 2016 
https://uberchromegw.corp.google.com/i/chromeos/builders/guado_moblab-paladin/builds/4271

Looks like a timeout

Suite job                 [ PASSED ]
moblab_DummyServerSuite   [ FAILED ]
moblab_DummyServerSuite     ABORT: Timed out, did not run.
Suite timings:
Downloads started at 2016-11-14 18:04:08
Payload downloads ended at 2016-11-14 18:04:16
Suite started at 2016-11-14 18:04:23
Artifact downloads ended (at latest) at 2016-11-14 18:04:25
Testing started at 2016-11-14 18:10:16
Testing ended at 2016-11-14 18:10:16
Links to test logs:
Suite job http://cautotest/tko/retrieve_logs.cgi?job=/results/85662766-chromeos-test/
moblab_DummyServerSuite http://cautotest/tko/retrieve_logs.cgi?job=/results/85662766-chromeos-test/
Attempting to display pool info: cq
host: chromeos2-row1-rack8-host1, status: Ready, locked: True diagnosis: Unused
labels: ['bluetooth', 'power:AC_only', 'storage:ssd', 'hw_video_acc_enc_h264', 'hw_jpeg_acc_dec', 'hw_video_acc_vp8', 'hw_video_acc_h264', 'board:guado_moblab', 'hw_video_acc_vp9', 'cts_abi_x86', 'cts_abi_arm', 'guado_moblab', 'pool:cq', 'sku:guado_intel_broadwell_i3_4Gb', 'variant:guado', 'os:moblab', 'phase:PVT', 'cros-version:guado_moblab-paladin/R56-8972.0.0-rc2']
Last 10 jobs within 2:18:00:
host: chromeos2-row1-rack8-host3, status: Ready, locked: False diagnosis: Unused
labels: ['bluetooth', 'power:AC_only', 'storage:ssd', 'hw_video_acc_enc_h264', 'hw_jpeg_acc_dec', 'hw_video_acc_vp8', 'hw_video_acc_h264', 'board:guado_moblab', 'hw_video_acc_vp9', 'cts_abi_x86', 'cts_abi_arm', 'guado_moblab', 'pool:cq', 'variant:guado', 'os:moblab', 'sku:guado_intel_broadwell_celeron_2Gb', 'phase:PVT', 'cros-version:guado_moblab-paladin/R56-8914.0.0-rc4']
Last 10 jobs within 2:18:00:
host: chromeos2-row2-rack8-host1, status: Repairing, locked: False diagnosis: Unused
labels: ['bluetooth', 'power:AC_only', 'storage:ssd', 'hw_jpeg_acc_dec', 'hw_video_acc_h264', 'board:guado_moblab', 'hw_video_acc_vp9', 'cts_abi_x86', 'cts_abi_arm', 'guado_moblab', 'pool:cq', 'os:moblab', 'sku:panther_intel_celeron_2Gb', 'phase:PVT2', 'variant:panther', 'cros-version:guado_moblab-paladin/R56-8990.0.0-rc1']
Last 10 jobs within 2:18:00:
127937 Provision started on: 2016-11-14 18:05:02 status FAIL
host: chromeos2-row2-rack8-host5, status: Ready, locked: False diagnosis: Unused
labels: ['bluetooth', 'power:AC_only', 'storage:ssd', 'hw_video_acc_enc_h264', 'hw_jpeg_acc_dec', 'hw_video_acc_vp8', 'hw_video_acc_h264', 'board:guado_moblab', 'hw_video_acc_vp9', 'cts_abi_x86', 'cts_abi_arm', 'guado_moblab', 'variant:guado', 'os:moblab', 'sku:guado_intel_broadwell_celeron_2Gb', 'phase:PVT', 'pool:cq', 'cros-version:guado_moblab-paladin/R56-8929.0.0-rc1']
Last 10 jobs within 2:18:00:
Reason: Some test(s) was aborted before running, suite must have timed out.
Output below this line is for buildbot consumption:
Will return from run_suite with status: SUITE_TIMEOUT

Sheriff investigating

Comment 1 by skau@chromium.org, Nov 15 2016 

No obvious CLs to blame.  Suspect that it might be a flake in moblab infrastructure.  Will wait on next CQ.

Comment 2 by ntang@google.com, Nov 15 2016

Cc: jrbarnette@chromium.org sbasi@chromium.org
Labels: OS-Chrome

Comment 3 by ntang@google.com, Nov 15 2016

Cc: pprabhu@chromium.org

Comment 4 by semenzato@chromium.org, Nov 15 2016

Cc: semenzato@chromium.org
It's not infra failure: the DUT is not responding.

guado_moblab-paladin build 4271

The HWTest stdio just says "timed out" and has a pointer to this bucket:

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/85662766-chromeos-test/hostless/debug/

The autoserv.DEBUG has this relevant information:

11/14 18:04:24.874 DEBUG|        base_utils:0185| Running 'ssh 100.115.245.200 'curl "http://100.115.245.200:8082/is_staged?artifacts=control_files,test_suites&files=&archive_url=gs://chromeos-image-archive/guado_moblab-paladin/R56-8990.0.0-rc2"''
11/14 18:04:25.922 DEBUG|        dev_server:0874| whether artifact is staged: 'True'
11/14 18:04:25.924 INFO |        dev_server:0993| Finished staging artifacts: build=guado_moblab-paladin/R56-8990.0.0-rc2, artifacts=['control_files', 'test_suites'], files=, archive_url=gs://chromeos-image-archive/guado_moblab-paladin/R56-8990.0.0-rc2
11/14 18:04:25.926 DEBUG|             suite:1145| Getting control file list for suite: moblab_quick
11/14 18:04:25.926 DEBUG|        base_utils:0185| Running 'ssh 100.115.245.200 'curl "http://100.115.245.200:8082/list_suite_controls?suite_name=moblab_quick&build=guado_moblab-paladin/R56-8990.0.0-rc2"''
11/14 18:04:27.066 DEBUG|             suite:1156| Parsing control files ...
11/14 18:04:27.069 DEBUG|             suite:1221| Parsed 1 control files.
11/14 18:04:27.069 DEBUG|             suite:0907| Discovered 1 stable tests.
11/14 18:04:27.069 DEBUG|             suite:0909| Discovered 0 unstable tests.
11/14 18:04:27.070 INFO |        server_job:0153| INFO	----	Start moblab_quick	timestamp=1479175467	localtime=Nov 14 18:04:27	
11/14 18:04:27.071 DEBUG|             suite:0825| Scheduling moblab_DummyServerSuite
11/14 18:04:27.496 DEBUG|             suite:1079| Adding job keyval for moblab_DummyServerSuite=85662767-chromeos-test
11/14 18:04:27.496 DEBUG|     dynamic_suite:0554| Waiting on suite.
11/14 18:10:16.514 INFO |        server_job:0153| START	----	guado_moblab-paladin/R56-8990.0.0-rc2/moblab_quick/moblab_DummyServerSuite	timestamp=1479175816	localtime=Nov 14 18:10:16	
11/14 18:10:16.515 INFO |        server_job:0153| 	ABORT	----	guado_moblab-paladin/R56-8990.0.0-rc2/moblab_quick/moblab_DummyServerSuite	timestamp=1479175816	localtime=Nov 14 18:10:16	
11/14 18:10:16.515 INFO |        server_job:0153| END ABORT	----	guado_moblab-paladin/R56-8990.0.0-rc2/moblab_quick/moblab_DummyServerSuite	timestamp=1479175816	localtime=Nov 14 18:10:16	
11/14 18:10:21.541 DEBUG|     dynamic_suite:0556| Finished waiting on suite. Returning from _perform_reimage_and_run.
11/14 18:10:21.541 DEBUG|     dynamic_suite:0495| Returning from dynamic_suite.reimage_and_run.

Fortunately I noticed the "Adding job keyval" with the job ID.  So I went to that bucket:

https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/85662767-chromeos-test/chromeos2-row2-rack8-host1/debug/

which shows that all connection attempts to the DUT failed.  I tried pinging the DUT and it appears to be dead.  This ought to be reported better.

Comment 5 by ntang@google.com, Nov 15 2016 

The test picked up chromeos2-row2-rack8-host1 and may caused the DUT to repair/fail state (http://cautotest/afe/#tab_id=view_host&object_id=1345). Similar case (DUT could not be repaired) seemed to happen in 665080. 

Do we know what kind of image is provisioned on the DUT? I assume it is an test image. However, since it is dead, I could not check it.

Comment 6 by semenzato@chromium.org, Nov 15 2016

Mergedinto: 662625
Status: Duplicate (was: Untriaged)
This DUT has been having trouble lately, there are three open bugs.  I am duplicating to the one that is getting the most attention.
►

Sign in to add a comment