New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 641312 link

Starred by 1 user

Issue metadata

Status: Archived
Owner:
Closed: Aug 2016
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 0
Type: Bug



Sign in to add a comment

guado_moblab: Repair image is too old

Project Member Reported by drinkcat@chromium.org, Aug 26 2016

Issue description

3 failures in a row, blocks CQ:
https://uberchromegw.corp.google.com/i/chromeos/builders/guado_moblab-paladin/builds/3590
https://uberchromegw.corp.google.com/i/chromeos/builders/guado_moblab-paladin/builds/3591
https://uberchromegw.corp.google.com/i/chromeos/builders/guado_moblab-paladin/builds/3592

https://uberchromegw.corp.google.com/i/chromeos/builders/guado_moblab-paladin/builds/3591/steps/HWTest%20%5Bmoblab_quick%5D/logs/stdio

  Attempting to display pool info: cq
  host: chromeos2-row1-rack8-host1, status: Repair Failed, locked: False diagnosis: Failed repair
  labels: ['bluetooth', 'power:AC_only', 'storage:ssd', 'hw_video_acc_enc_h264', 'hw_jpeg_acc_dec', 'hw_video_acc_vp8', 'hw_video_acc_h264', 'board:guado_moblab', 'hw_video_acc_vp9', 'cts_abi_x86', 'cts_abi_arm', 'guado_moblab', 'pool:cq']
  Last 10 jobs within 2:18:00:
  72790 Repair started on: 2016-08-26 04:53:25 status FAIL
  72779 Repair started on: 2016-08-26 04:38:27 status FAIL
  72777 Verify started on: 2016-08-26 04:37:55 status FAIL
  72754 Repair started on: 2016-08-26 04:08:15 status FAIL
  72751 Verify started on: 2016-08-26 04:07:47 status FAIL
  72725 Repair started on: 2016-08-26 03:38:03 status FAIL
  72721 Verify started on: 2016-08-26 03:37:36 status FAIL
  72673 Repair started on: 2016-08-26 03:07:55 status FAIL
  72669 Verify started on: 2016-08-26 03:07:29 status FAIL
  
  host: chromeos2-row1-rack8-host3, status: Repair Failed, locked: False diagnosis: Failed repair
  labels: ['bluetooth', 'power:AC_only', 'storage:ssd', 'hw_video_acc_enc_h264', 'hw_jpeg_acc_dec', 'hw_video_acc_vp8', 'hw_video_acc_h264', 'board:guado_moblab', 'hw_video_acc_vp9', 'cts_abi_x86', 'cts_abi_arm', 'guado_moblab', 'pool:cq', 'variant:guado', 'os:moblab', 'sku:guado_intel_broadwell_celeron_2Gb', 'phase:PVT']
  Last 10 jobs within 2:18:00:
  72787 Repair started on: 2016-08-26 04:52:51 status FAIL
  72778 Repair started on: 2016-08-26 04:38:22 status FAIL
  72775 Verify started on: 2016-08-26 04:37:55 status FAIL
  72753 Repair started on: 2016-08-26 04:08:10 status FAIL
  72752 Verify started on: 2016-08-26 04:07:47 status FAIL
  72724 Repair started on: 2016-08-26 03:37:59 status FAIL
  72722 Verify started on: 2016-08-26 03:37:36 status FAIL
  72672 Repair started on: 2016-08-26 03:07:51 status FAIL
  72667 Verify started on: 2016-08-26 03:07:29 status FAIL
  
  host: chromeos2-row2-rack8-host1, status: Ready, locked: True diagnosis: Unused
  labels: ['bluetooth', 'power:AC_only', 'storage:ssd', 'hw_video_acc_enc_h264', 'hw_jpeg_acc_dec', 'hw_video_acc_vp8', 'hw_video_acc_h264', 'board:guado_moblab', 'hw_video_acc_vp9', 'cts_abi_x86', 'cts_abi_arm', 'guado_moblab', 'pool:cq', 'cros-version:guado_moblab-paladin/R54-8686.0.0-rc2']
  Last 10 jobs within 2:18:00:
  
  Reason: Suite job failed or provisioning failed.

---

Side question: Why do we have so few guado_moblab in the lab, but still allow it to block CQ?
 

Comment 1 by sbasi@chromium.org, Aug 26 2016

Summary: guado_moblab: Repair image is too old (was: guado_moblab: HWTest failure (no usable device in pool:cq))
DownloaderException: Could not find *_full_* in Google Storage at gs://chromeos-image-archive/guado_moblab-release/R49-7825.0.0

Lots of DUTs are down. I moved from pool:bvt to pool:cq.

Don can you find a good repair image and kick off repair on these guys.

For the side question: Blocking on moblab in the cq has been proven to catch A LOT of bugs that would cause lab breakages/outages. I've asked for more space in the lab to install more setups but we're blocked on the deployment to the next lab which should begin hopefully soon.

Comment 2 by sbasi@chromium.org, Aug 26 2016

Cc: nxia@chromium.org
+NingNing in case Don is too busy but this is a P0
How is the expected image updated?

PS: In future, can we start using stable channel images from chromeos-releases for this? That would stop this being a panic updated every so often.

Better yet, can let these devices self-update via Omaha? That would let us manage this just like other ChromeOS releases, and there is a fully flushed work and actively maintained process that's free for us to use.

Comment 5 by sbasi@chromium.org, Aug 26 2016

Cc: akes...@chromium.org
As to your suggestions, lets flesh that out into a 1 paragraph description/approach and assign a noogler.

+Aviv for noogler project visibility.
Updated to R54-8743.0.0.

Repair jobs scheduled for the duts:
chromeos2-row1-rack8-host1
chromeos2-row1-rack8-host3
chromeos2-row2-rack8-host1
chromeos2-row2-rack8-host7
chromeos2-row2-rack8-host11


Status: Fixed (was: Assigned)
Believed to be fixed.
#1: Thanks for the answer to the side question ,-)

Comment 9 by dchan@chromium.org, Oct 7 2016

Labels: VerifyIn-55

Comment 10 by dchan@chromium.org, Oct 10 2016

Labels: -VerifyIn-55

Comment 11 by dchan@google.com, Nov 19 2016

Labels: VerifyIn-56

Comment 12 by dchan@google.com, Jan 21 2017

Labels: VerifyIn-57

Comment 13 by dchan@google.com, Mar 4 2017

Labels: VerifyIn-58

Comment 14 by dchan@google.com, Apr 17 2017

Labels: VerifyIn-59

Comment 15 by dchan@google.com, May 30 2017

Labels: VerifyIn-60
Labels: VerifyIn-61

Comment 17 by dchan@chromium.org, Oct 14 2017

Status: Archived (was: Fixed)

Sign in to add a comment