moblab-vm-generic-paladin: host did not return from reboot |
||||||
Issue descriptionThis broke a CQ run. This build failed because (I think) some provision step in moblab-vm autotests did not return from reboot. The only other error was on gale-paladin which was set as experimental in the tree status. https://ci.chromium.org/p/chromeos/builders/luci.chromeos.general/Prod/b8924093670230016256 I found a CIDB failure in this log, but I don't know if it's related. https://logs.chromium.org/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8924093670230016256/+/steps/CommitQueueCompletion/0/stdout 04:43:00: INFO: Build config moblab-generic-vm-paladin completed: CIDB status: fail. Buildbucket status STARTED result None. Then I followed the moblab-vm-generic-paladin link and found this: https://ci.chromium.org/p/chromeos/builders/luci.chromeos.general/CQ/b8924092351205464368 and from there I found this moblab test log: https://logs.chromium.org/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8924092351205464368/+/steps/MoblabVMTest/0/stdout which first shows an error getting the HWIDLabel: 04:16:34 INFO | autoserv| Retrying in 13.670080 seconds... 04:17:00 INFO | autoserv| Setting image_storage_server to /mnt/moblab/static/prefetched/ 04:17:09 INFO | autoserv| [stderr] ERROR:root:error getting label HWIDLabel. and shortly later: 04:34:11 INFO | autoserv| [stderr] 01-17-2019 [04:34:11] Suite job is finished. 04:34:11 INFO | autoserv| [stderr] 01-17-2019 [04:34:11] Start collecting test results and dump them to json. 04:34:12 INFO | autoserv| [stderr] Suite job [ PASSED ] 04:34:12 INFO | autoserv| [stderr] provision [ FAILED ] 04:34:12 INFO | autoserv| [stderr] provision ABORT: Host did not return from reboot I am guessing that's the problem.
,
Yesterday
(45 hours ago)
Keep seeing these error messages on moblab-vm-generic-paladin builder which keeps blocking CQ. Marking moblab-vm-generic-paladin to experimental seems not work. Raised the priority to 0.
,
Yesterday
(44 hours ago)
,
Today
(14 hours ago)
Isn't it the case that moblab-vm-generic-paladin could not break the CQ because it was set as EXPERIMENTAL in http://chromiumos-status.appspot.com?
,
Today
(14 hours ago)
Please correct me if I am wrong. Also, I saw a CL which makes it experimental permanently.
,
Today
(13 hours ago)
The CL is https://crrev.com/c/1424718 Also, I just noticed the comments on crosoncall which state that EXPERIMENTAL= doesn't always work. (Do we have a bug for that?)
,
Today
(13 hours ago)
> Also, I just noticed the comments on crosoncall which state that EXPERIMENTAL= doesn't always work. Can you clarify your expectations here? The only place that EXPERIMENTAL= has an effect is in the status at chromiumos-status.appspot.com
,
Today
(13 hours ago)
Never mind, I misunderstood the comment. I have read the crosoncall thread. I'm not aware of the EXPERIMENTAL= mechanism being flaky per se except that the code that makes it work is quite strict about the format of the status message. If you believe there is an instance of it not working then please open a bug.
,
Today
(12 hours ago)
Ohh, sorry for being unclear. Originally, I thought the EXPERIMENTAL= not work since the builder does not show experimental mark "*" next to their name in legoland. But finally I found out that only the builder being set to experimental permanently will have "*" mark. And the EXPERIMENTAL= actually works on the build result. Currently, it is not moblab-vm-generic-paladin block the CQ, but it keeps failing in latest builds. Please also help to suggest if you know how we can solve the error. The logs are: https://luci-logdog.appspot.com/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8923642069053448480/+/steps/MoblabVMTest/0/stdout If you think it would be better to create a new bug for this issue, please let me know. Thanks.
,
Today
(12 hours ago)
And I think we can remove moblab-vm-generic-paladin from being experimental on tree status and revert the CL which makes it experimental once we make those failing tests passed.
,
Today
(12 hours ago)
#9 thank you for clarifying. Please look for existing moblab-vm-generic bugs that may be covering the issue you're seeing, and if you don't find one, or are not sure, open another bug. |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by jclinton@google.com
, Jan 17 (5 days ago)Owner: haddowk@chromium.org
Status: Assigned (was: Untriaged)