provision failing on skylab DUTs due to quick_provision failure |
||||
Issue descriptionExample failed task: https://chrome-swarming.appspot.com/task?id=3e99c690ce599710
,
Jul 9
Looks like the DUT doesn't have wget?
07/09 14:36:51.962 DEBUG| utils:0286| [stdout] 2018-07-09 14:36:51-07:00 INFO: Update kernel /dev/mmcblk1p4
07/09 14:36:51.962 DEBUG| utils:0286| [stdout] 2018-07-09 14:36:51-07:00 INFO: Updated status: DUT: Updating kernel /dev/mmcblk1p4
07/09 14:36:51.962 DEBUG| utils:0286| [stdout] 2018-07-09 14:36:51-07:00 INFO: Updating /dev/mmcblk1p4 with http://100.115.185.227:8082/static/nyan_blaze-release/R69-10763.0.0/full_dev_part_KERN.bin.gz
07/09 14:36:52.021 DEBUG| utils:0286| [stdout] /tmp/quick-provision: line 133: /usr/local/bin/wget: cannot execute binary file: Exec format error
07/09 14:36:52.021 DEBUG| utils:0286| [stdout]
07/09 14:36:52.021 DEBUG| utils:0286| [stdout] gzip: stdin: unexpected end of file
07/09 14:36:52.021 DEBUG| utils:0286| [stdout] 0+0 records in
07/09 14:36:52.021 DEBUG| utils:0286| [stdout] 0+0 records out
07/09 14:36:52.022 DEBUG| utils:0286| [stdout] 0 bytes copied, 0.00509399 s, 0.0 kB/s
07/09 14:36:52.022 DEBUG| utils:0286| [stdout] 2018-07-09 14:36:51-07:00 ERROR: FATAL: Decompressing http://100.115.185.227:8082/static/nyan_blaze-release/R69-10763.0.0/full_dev_part_KERN.bin.gz failed. (statuses 0 1 0)
07/09 14:36:52.022 DEBUG| utils:0286| [stdout] 2018-07-09 14:36:51-07:00 INFO: Updated status: FATAL: Decompressing http://100.115.185.227:8082/static/nyan_blaze-release/R69-10763.0.0/full_dev_part_KERN.bin.gz failed. (statuses 0 1 0)
07/09 14:36:52.023 ERROR| autoupdater:0854| quick-provision script failed; will fall back to update_engine.
Traceback (most recent call last):
File "/usr/local/autotest/server/cros/autoupdater.py", line 847, in _install_via_quick_provision
self._run(command)
File "/usr/local/autotest/server/cros/autoupdater.py", line 325, in _run
return self.host.run(cmd, *args, **kwargs)
File "/usr/local/autotest/server/hosts/ssh_host.py", line 323, in run
return self.run_very_slowly(*args, **kwargs)
File "/usr/local/autotest/server/hosts/ssh_host.py", line 312, in run_very_slowly
ssh_failure_retry_ok)
File "/usr/local/autotest/server/hosts/ssh_host.py", line 262, in _run
raise error.AutoservRunError("command execution error", result)
,
Jul 9
Something is very whacky with that DUT: localhost /usr/local/bin # ./wget -bash: ./wget: cannot execute binary file: Exec format error localhost /usr/local/bin # ./file /usr/local/bin/file.bin: 1: /usr/local/bin/file.bin: Syntax error: word unexpected (expecting ")")
,
Jul 9
Installed version: GOOGLE_RELEASE=10828.0.0-rc3 CHROMEOS_RELEASE_BOARD=reef Repair image: R68-10718.34.0
,
Jul 10
> Example failed task: https://chrome-swarming.appspot.com/task?id=3e99c690ce599710 From that link, I'm unable to find links to the test logs. Also, I need the ability to run dut-status (or at least, to see the full DUT history, sorted by time with both tests and special tasks).
,
Jul 10
> From that link, I'm unable to find links to the test logs. Specifically, I mean the logs from autoserv. Also, nowhere is the DUT identified (although, to be fair, that information would be in the autoserv logs).
,
Jul 10
Re #5: The milo logs now do point to the gs_offloader offloaded logs. See "results" in https://chrome-swarming.appspot.com/task?id=3e99c690ce599710 Re #6: dut-history tool as such is not yet available for skylab unfortunately. But a single dut/bot's history is available: Click through to "bot assigned to task": https://chrome-swarming.appspot.com/bot?id=chromeos-skylab-bot-cbece344-04a4-463a-b440-7d105b6cc845&sort_stats=total%3Adesc The DUT is identified in the dimension called "dut_name". I've wanted to make the dut_name the id of the bot but didn't get around to it for logistical reasons.
,
Jul 10
Initiated a power wash on the dut by: echo 'clobber' >/mnt/stateful_partition/.update_available reboot And retrying provision: https://chrome-swarming.appspot.com/task?id=3e9a5ab90b64b210
,
Jul 10
> Re #5: The milo logs now do point to the gs_offloader offloaded logs. See > "results" in https://chrome-swarming.appspot.com/task?id=3e99c690ce599710 More specifically, you must click the "Milo Output" link on the tab, then click on "results".
,
Jul 10
> Installed version: > GOOGLE_RELEASE=10828.0.0-rc3 > CHROMEOS_RELEASE_BOARD=reef > > Repair image: R68-10718.34.0 The logs from the failure have "nyan_blaze" written all over them, not "reef". The bot itself claims to be "reef".
,
Jul 10
ah, I hard-code a nyan_blaze release image to all DUTs. I will change that first.
,
Jul 10
Re #9: You should land on the milo UI by default. Don't you?
,
Jul 10
,
Jul 11
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/chromite/+/a8ec211e86abbdd18b829379b4d2f9ccac545dd4 commit a8ec211e86abbdd18b829379b4d2f9ccac545dd4 Author: Xixuan Wu <xixuan@chromium.org> Date: Wed Jul 11 19:13:06 2018 Skylab-launch: Remove hard-coded build for SkylabHWTestStage. BUG= chromium:861971 TEST=Ran unittest. Ran tryjob. Change-Id: I5e63c5f9772b7c6cfc273308b18d32db343add92 Reviewed-on: https://chromium-review.googlesource.com/1131693 Commit-Ready: Xixuan Wu <xixuan@chromium.org> Tested-by: Xixuan Wu <xixuan@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/a8ec211e86abbdd18b829379b4d2f9ccac545dd4/cbuildbot/stages/test_stages.py [modify] https://crrev.com/a8ec211e86abbdd18b829379b4d2f9ccac545dd4/cbuildbot/builders/simple_builders.py
,
Jul 13
|
||||
►
Sign in to add a comment |
||||
Comment 1 by xixuan@chromium.org
, Jul 9