New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 861971 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jul 13
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 1
Type: Bug



Sign in to add a comment

provision failing on skylab DUTs due to quick_provision failure

Project Member Reported by pprabhu@chromium.org, Jul 9

Issue description

Cc: jrbarnette@chromium.org
Looks like the DUT doesn't have wget?

07/09 14:36:51.962 DEBUG|             utils:0286| [stdout] 2018-07-09 14:36:51-07:00 INFO: Update kernel /dev/mmcblk1p4
07/09 14:36:51.962 DEBUG|             utils:0286| [stdout] 2018-07-09 14:36:51-07:00 INFO: Updated status: DUT: Updating kernel /dev/mmcblk1p4
07/09 14:36:51.962 DEBUG|             utils:0286| [stdout] 2018-07-09 14:36:51-07:00 INFO: Updating /dev/mmcblk1p4 with http://100.115.185.227:8082/static/nyan_blaze-release/R69-10763.0.0/full_dev_part_KERN.bin.gz
07/09 14:36:52.021 DEBUG|             utils:0286| [stdout] /tmp/quick-provision: line 133: /usr/local/bin/wget: cannot execute binary file: Exec format error
07/09 14:36:52.021 DEBUG|             utils:0286| [stdout] 
07/09 14:36:52.021 DEBUG|             utils:0286| [stdout] gzip: stdin: unexpected end of file
07/09 14:36:52.021 DEBUG|             utils:0286| [stdout] 0+0 records in
07/09 14:36:52.021 DEBUG|             utils:0286| [stdout] 0+0 records out
07/09 14:36:52.022 DEBUG|             utils:0286| [stdout] 0 bytes copied, 0.00509399 s, 0.0 kB/s
07/09 14:36:52.022 DEBUG|             utils:0286| [stdout] 2018-07-09 14:36:51-07:00 ERROR: FATAL: Decompressing http://100.115.185.227:8082/static/nyan_blaze-release/R69-10763.0.0/full_dev_part_KERN.bin.gz failed. (statuses 0 1 0)
07/09 14:36:52.022 DEBUG|             utils:0286| [stdout] 2018-07-09 14:36:51-07:00 INFO: Updated status: FATAL: Decompressing http://100.115.185.227:8082/static/nyan_blaze-release/R69-10763.0.0/full_dev_part_KERN.bin.gz failed. (statuses 0 1 0)
07/09 14:36:52.023 ERROR|       autoupdater:0854| quick-provision script failed; will fall back to update_engine.
Traceback (most recent call last):
  File "/usr/local/autotest/server/cros/autoupdater.py", line 847, in _install_via_quick_provision
    self._run(command)
  File "/usr/local/autotest/server/cros/autoupdater.py", line 325, in _run
    return self.host.run(cmd, *args, **kwargs)
  File "/usr/local/autotest/server/hosts/ssh_host.py", line 323, in run
    return self.run_very_slowly(*args, **kwargs)
  File "/usr/local/autotest/server/hosts/ssh_host.py", line 312, in run_very_slowly
    ssh_failure_retry_ok)
  File "/usr/local/autotest/server/hosts/ssh_host.py", line 262, in _run
    raise error.AutoservRunError("command execution error", result)
Something is very whacky with that DUT:

localhost /usr/local/bin # ./wget
-bash: ./wget: cannot execute binary file: Exec format error
localhost /usr/local/bin # ./file
/usr/local/bin/file.bin: 1: /usr/local/bin/file.bin: Syntax error: word unexpected (expecting ")")
Installed version:
GOOGLE_RELEASE=10828.0.0-rc3
CHROMEOS_RELEASE_BOARD=reef

Repair image: R68-10718.34.0

> Example failed task: https://chrome-swarming.appspot.com/task?id=3e99c690ce599710

From that link, I'm unable to find links to the test logs.
Also, I need the ability to run dut-status (or at least, to
see the full DUT history, sorted by time with both tests and
special tasks).

> From that link, I'm unable to find links to the test logs.

Specifically, I mean the logs from autoserv.

Also, nowhere is the DUT identified (although, to be fair,
that information would be in the autoserv logs).

Re #5: The milo logs now do point to the gs_offloader offloaded logs. See 
 "results" in https://chrome-swarming.appspot.com/task?id=3e99c690ce599710

Re #6: dut-history tool as such is not yet available for skylab unfortunately. But a single dut/bot's history is available: Click through to "bot assigned to task": https://chrome-swarming.appspot.com/bot?id=chromeos-skylab-bot-cbece344-04a4-463a-b440-7d105b6cc845&sort_stats=total%3Adesc

The DUT is identified in the dimension called "dut_name". I've wanted to make the dut_name the id of the bot but didn't get around to it for logistical reasons.
Initiated a power wash on the dut by:

echo 'clobber' >/mnt/stateful_partition/.update_available
reboot

And retrying provision: https://chrome-swarming.appspot.com/task?id=3e9a5ab90b64b210
> Re #5: The milo logs now do point to the gs_offloader offloaded logs. See 
>  "results" in https://chrome-swarming.appspot.com/task?id=3e99c690ce599710

More specifically, you must click the "Milo Output" link on the tab,
then click on "results".

> Installed version:
> GOOGLE_RELEASE=10828.0.0-rc3
> CHROMEOS_RELEASE_BOARD=reef
> 
> Repair image: R68-10718.34.0

The logs from the failure have "nyan_blaze" written all over them,
not "reef".

The bot itself claims to be "reef".

Owner: xixuan@chromium.org
ah, I hard-code a nyan_blaze release image to all DUTs. I will change that first.
Re #9: You should land on the milo UI by default. Don't you?
Labels: -Pri-3 Pri-1
Project Member

Comment 14 by bugdroid1@chromium.org, Jul 11

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/a8ec211e86abbdd18b829379b4d2f9ccac545dd4

commit a8ec211e86abbdd18b829379b4d2f9ccac545dd4
Author: Xixuan Wu <xixuan@chromium.org>
Date: Wed Jul 11 19:13:06 2018

Skylab-launch: Remove hard-coded build for SkylabHWTestStage.

BUG= chromium:861971 
TEST=Ran unittest. Ran tryjob.

Change-Id: I5e63c5f9772b7c6cfc273308b18d32db343add92
Reviewed-on: https://chromium-review.googlesource.com/1131693
Commit-Ready: Xixuan Wu <xixuan@chromium.org>
Tested-by: Xixuan Wu <xixuan@chromium.org>
Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org>

[modify] https://crrev.com/a8ec211e86abbdd18b829379b4d2f9ccac545dd4/cbuildbot/stages/test_stages.py
[modify] https://crrev.com/a8ec211e86abbdd18b829379b4d2f9ccac545dd4/cbuildbot/builders/simple_builders.py

Status: Fixed (was: Started)

Sign in to add a comment