New issue
Advanced search Search tips
Note: Color blocks (like █ or █) mean that a user may not be available. Tooltip shows the reason.

Issue 615664 link

Starred by 1 user
Status: Fixed
Owner:
xixuan@chromium.org
Closed: Dec 2016
Cc:
█ jrbarnette@chromium.org
█ kevcheng@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 3
Type: Bug



Sign in to add a comment

cyan-cheets-paladin HWTest [arc-bvt-cq] abort

Project Member Reported by kevcheng@chromium.org, May 28 2016 
https://uberchromegw.corp.google.com/i/chromeos/builders/cyan-cheets-paladin/builds/593

I think there were too few duts (looked like only 3 working duts) in the cq pool causing the suite to time out:

Suite timings:
Testing started at 2016-05-21 04:47:37
Testing ended at 2016-05-21 06:15:49

Attempting to diagnose pool: cq
host: chromeos4-row6-rack9-host12, status: Running, locked: False diagnosis: Working
labels: ['bluetooth', 'hw_video_acc_enc_h264', 'hw_jpeg_acc_dec', 'power:battery', 'ec:cros', 'hw_video_acc_vp8', 'hw_video_acc_vp9', 'cyan', 'hw_video_acc_h264', 'storage:mmc', 'webcam', 'board:cyan-cheets', 'pool:cq', 'internal_display', 'audio_loopback_dongle', 'os:cros', 'servo', 'phase:PVT', 'touchpad', 'touchscreen', 'sku:cyan_intel_celeron_n3150_4Gb', 'variant:cyan', 'accel:cros-ec', 'cros-version:cyan-cheets-paladin/R53-8354.0.0-rc2']
Last 10 jobs within 1:48:00:
55599362 Reset started on: 2016-05-21 06:17:16 status PASS
64431477 cyan-cheets-paladin/R53-8354.0.0-rc2/arc-bvt-cq/video_VideoSanity.vp9 started on: 2016-05-21 06:12:27 status Completed
55599089 Reset started on: 2016-05-21 06:10:06 status PASS
64431471 cyan-cheets-paladin/R53-8354.0.0-rc2/arc-bvt-cq/desktopui_ExitOnSupervisedUserCrash started on: 2016-05-21 06:04:16 status Completed
55598835 Reset started on: 2016-05-21 06:01:58 status PASS
64431440 cyan-cheets-paladin/R53-8354.0.0-rc2/arc-bvt-cq/cheets_Notification started on: 2016-05-21 05:57:13 status Completed
55598550 Reset started on: 2016-05-21 05:55:10 status PASS
64431446 cyan-cheets-paladin/R53-8354.0.0-rc2/arc-bvt-cq/video_ChromeHWDecodeUsed.vp9 started on: 2016-05-21 05:51:13 status Completed
55597765 Provision started on: 2016-05-21 05:31:51 status PASS
64429980 cyan-cheets-android-pfq/R53-8354.0.0-rc1/arc-bvt-cq/video_VideoSanity.vp9 started on: 2016-05-21 05:17:54 status Completed

host: chromeos4-row6-rack9-host20, status: Ready, locked: False diagnosis: Working
labels: ['bluetooth', 'touchpad', 'touchscreen', 'ec:cros', 'audio_loopback_dongle', 'hw_jpeg_acc_dec', 'power:battery', 'hw_video_acc_vp8', 'hw_video_acc_enc_h264', 'hw_video_acc_vp9', 'cyan', 'hw_video_acc_h264', 'storage:mmc', 'webcam', 'board:cyan-cheets', 'pool:cq', 'internal_display', 'os:cros', 'phase:PVT', 'sku:cyan_intel_celeron_n3150_4Gb', 'variant:cyan', 'servo', 'accel:cros-ec', 'cros-version:cyan-cheets-paladin/R53-8354.0.0-rc2']
Last 10 jobs within 1:48:00:
64431443 cyan-cheets-paladin/R53-8354.0.0-rc2/arc-bvt-cq/video_ChromeRTCHWDecodeUsed started on: 2016-05-21 06:13:15 status Completed
55599121 Reset started on: 2016-05-21 06:10:52 status PASS
64431473 cyan-cheets-paladin/R53-8354.0.0-rc2/arc-bvt-cq/video_VideoSanity.vp8 started on: 2016-05-21 06:05:21 status Completed
55598872 Reset started on: 2016-05-21 06:02:59 status PASS
64431459 cyan-cheets-paladin/R53-8354.0.0-rc2/arc-bvt-cq/cheets_SettingsBridge started on: 2016-05-21 05:58:15 status Completed
55598597 Reset started on: 2016-05-21 05:56:03 status PASS
64431432 cyan-cheets-paladin/R53-8354.0.0-rc2/arc-bvt-cq/telemetry_LoginTest started on: 2016-05-21 05:51:42 status Completed
55597794 Provision started on: 2016-05-21 05:32:30 status PASS
64429979 cyan-cheets-android-pfq/R53-8354.0.0-rc1/arc-bvt-cq/video_VideoSanity.vp8 started on: 2016-05-21 05:17:54 status Completed
55597249 Reset started on: 2016-05-21 05:15:49 status PASS

host: chromeos4-row6-rack9-host18, status: Ready, locked: False diagnosis: Working
labels: ['bluetooth', 'touchpad', 'board:cyan-cheets', 'ec:cros', 'audio_loopback_dongle', 'hw_jpeg_acc_dec', 'power:battery', 'hw_video_acc_vp8', 'hw_video_acc_enc_h264', 'hw_video_acc_vp9', 'cyan', 'hw_video_acc_h264', 'storage:mmc', 'webcam', 'pool:cq', 'internal_display', 'touchscreen', 'os:cros', 'phase:PVT', 'sku:cyan_intel_celeron_n3150_4Gb', 'variant:cyan', 'servo', 'accel:cros-ec', 'cros-version:cyan-cheets-paladin/R53-8354.0.0-rc2']
Last 10 jobs within 1:48:00:
64431464 cyan-cheets-paladin/R53-8354.0.0-rc2/arc-bvt-cq/video_ChromeCameraMJpegHWDecodeUsed started on: 2016-05-21 06:12:27 status Completed
55599088 Reset started on: 2016-05-21 06:10:05 status PASS
64431466 cyan-cheets-paladin/R53-8354.0.0-rc2/arc-bvt-cq/graphics_Idle.arc started on: 2016-05-21 06:04:16 status Completed
55598834 Reset started on: 2016-05-21 06:01:57 status PASS
64431454 cyan-cheets-paladin/R53-8354.0.0-rc2/arc-bvt-cq/cheets_KeyboardTest started on: 2016-05-21 05:57:13 status Completed
55598551 Reset started on: 2016-05-21 05:55:10 status PASS
64431436 cyan-cheets-paladin/R53-8354.0.0-rc2/arc-bvt-cq/video_VideoSanity.h264 started on: 2016-05-21 05:51:13 status Completed
55598371 Reset started on: 2016-05-21 05:49:27 status PASS
64431434 cyan-cheets-paladin/R53-8354.0.0-rc2/arc-bvt-cq/cheets_ContainerSmokeTest started on: 2016-05-21 05:45:19 status Completed
55598150 Reset started on: 2016-05-21 05:43:10 status PASS

host: chromeos4-row6-rack9-host16, status: Repair Failed, locked: False diagnosis: Failed repair
labels: ['bluetooth', 'touchpad', 'board:cyan-cheets', 'ec:cros', 'audio_loopback_dongle', 'hw_jpeg_acc_dec', 'power:battery', 'hw_video_acc_vp8', 'hw_video_acc_enc_h264', 'hw_video_acc_vp9', 'cyan', 'hw_video_acc_h264', 'storage:mmc', 'webcam', 'pool:cq', 'internal_display', 'touchscreen', 'os:cros', 'phase:PVT', 'sku:cyan_intel_celeron_n3150_4Gb', 'variant:cyan', 'accel:cros-ec']
Last 10 jobs within 1:48:00:
55598987 Repair started on: 2016-05-21 06:06:07 status FAIL
55598711 Verify started on: 2016-05-21 05:59:01 status FAIL
55596447 Repair started on: 2016-05-21 04:51:52 status FAIL

Comment 1 by dgarr...@chromium.org, May 31 2016

Cc: jrbarnette@chromium.org
Yes, we do have a shortage in these duts, cause unknown. b/29050699 covers the manual repairs.

I don't yet know why they are broken, or why the servos aren't doing automatic repair.

Comment 2 by jrbarnette@chromium.org, May 31 2016 

I don't know why the cyans went down, but as for why servo
isn't repairing them, that's pretty easy:
dut-status -n -b cyan-cheets | xargs servo-stat 
chromeos4-row6-rack9-host1 ...A is down
chromeos4-row6-rack9-host2 ...A is down
chromeos4-row6-rack9-host5 ...A is down
chromeos4-row6-rack9-host6 ...A is down
chromeos4-row6-rack9-host9 ...A is down
chromeos4-row6-rack9-host11 ...A is down
chromeos4-row6-rack9-host10 ...A is down
chromeos4-row6-rack9-host18 ...A is down
chromeos4-row6-rack9-host16 ...A is down

This is probably an aftereffect of the recent servo outage.

Comment 3 by jrbarnette@chromium.org, May 31 2016 

... and really, someone should look at the history of the failures,
because there's a decent chance that the DUTs are down because of
a Chrome OS or cheets bug.

Comment 4 by autumn@chromium.org, Jun 2 2016

Cc: -dgarr...@chromium.org
Owner: dgarr...@chromium.org
+ don to check current inventory

Comment 5 by dgarr...@chromium.org, Jun 6 2016

Owner: xixuan@chromium.org
There were multiple fire team bugs to fix these duts, but I think the problem has now been addressed. Passing to current deputy to confirm.

Comment 6 by xixuan@chromium.org, Dec 9 2016

Status: Fixed (was: Untriaged)
bet this issue has been fixed since it's from Jun...
►

Sign in to add a comment