New issue
Advanced search Search tips

Issue 873351 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Aug 14
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

power.Reboot test fails due to race when DUT reboots quickly

Project Member Reported by derat@chromium.org, Aug 10

Issue description

In the gnawty-release build at http://stainless/browse/chromeos-autotest-results/225399103-chromeos-test/, the power.Reboot test failed due to the reboot command timing out:

2018/08/09 09:15:38 Started test power.Reboot
2018/08/09 09:15:38 [09:15:38.642] Rebooting DUT
2018/08/09 09:20:38 [09:20:38.643] Error at reboot.go:27: Failed to reboot DUT: context deadline exceeded

The messages log shows that the system actually rebooted, though:

...
2018-08-09T09:15:39.375154-07:00 NOTICE pre-shutdown[12372]: Shutting down for reboot: not-via-powerd
...
2018-08-09T09:15:39.397593-07:00 ERR wpa_supplicant[587]: nl80211: Failed to set IPv4 unicast in multicast filter
2018-08-09T09:15:45.216842-07:00 INFO kernel: [    0.000000] Initializing cgroup subsys cpuset
2018-08-09T09:15:45.216914-07:00 INFO kernel: [    0.000000] Initializing cgroup subsys cpu
2018-08-09T09:15:45.216918-07:00 INFO kernel: [    0.000000] Initializing cgroup subsys cpuacct
2018-08-09T09:15:45.216920-07:00 NOTICE kernel: [    0.000000] Linux version 4.4.146-14675-gf9e8bb4004dd (chrome-bot@swarm-cros-464) (Chromium OS 7.0_pre331547_p20180529-r9 clang version 7.0.0 (/var/cache/chromeos-cache/distfiles/host/egit-src/clang.git 1084e053422cbbaaa6e06a972aac210489e7a0ad) (/var/cache/chromeos-cache/distfiles/host/egit-src/llvm.git 
...

I think that my code in power.Reboot to run the "reboot" command is buggy. It attempts to run a subshell that sleeps before rebooting (so the SSH exec request will finish successfully), but it actually blocks until the reboot command returns, resulting in a race where the command will appear to hang if the system reboots too quickly.
 
Project Member

Comment 1 by bugdroid1@chromium.org, Aug 14

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/7aef29de8a3f5786b5f7cedfc88a7c8b408572aa

commit 7aef29de8a3f5786b5f7cedfc88a7c8b408572aa
Author: Daniel Erat <derat@chromium.org>
Date: Tue Aug 14 09:13:39 2018

tast-tests: Fix hanging-command race in power.Reboot.

Make the power.Reboot test correctly perform a delayed
reboot in the background using nohup and redirecting stdout,
stderr, and stdin. The old implementation just ran "(sleep
1; reboot) &", which blocks until I/O streams are closed.

BUG= chromium:873351 
TEST=manual: test still passes, but no delay between
     "Rebooting DUT" and "Waiting for DUT to become
     unreachable" messages now

Change-Id: I4db458b9cd42a716cfbbc62374c3a93d92a2f2a8
Reviewed-on: https://chromium-review.googlesource.com/1171812
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[modify] https://crrev.com/7aef29de8a3f5786b5f7cedfc88a7c8b408572aa/src/chromiumos/tast/remote/bundles/cros/power/reboot.go

Status: Fixed (was: Started)

Sign in to add a comment