New issue
Advanced search Search tips

Issue 859212 link

Starred by 1 user

Issue metadata

Status: Fixed
Owner:
Closed: Jul 2
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

platform.CheckProcesses fails due to debugd restart

Project Member Reported by derat@chromium.org, Jun 29 2018

Issue description

The platform.CheckProcesses Tast test is failing almost every time when it runs on release builders now due to a missing debugd process:

2018/06/28 10:13:06 Started test platform.CheckProcesses
2018/06/28 10:13:06 [10:13:05.715] conntrackd is running
2018/06/28 10:13:06 [10:13:05.715] dbus-daemon is running
2018/06/28 10:13:06 [10:13:05.715] Error at check_processes.go:59: debugd not running
2018/06/28 10:13:06 [10:13:05.715] Stack trace:
...
2018/06/28 10:13:06 [10:13:05.715] metrics_daemon is running
2018/06/28 10:13:06 [10:13:05.715] powerd is running
2018/06/28 10:13:06 [10:13:05.715] shill is running
2018/06/28 10:13:06 [10:13:05.716] udevd is running
2018/06/28 10:13:06 [10:13:05.716] tlsdated is running
2018/06/28 10:13:06 [10:13:05.716] update_engine is running
2018/06/28 10:13:06 [10:13:05.716] wpa_supplicant is running
2018/06/28 10:13:06 Completed test platform.CheckProcesses in 36ms with 1 error(s)

The reason for this is somewhat interesting: this test typically runs shortly after example.DBus, which I recently changed to restart the session_manager job rather than the powerd job ( issue 856390 ).

debugd's Upstart job contains this:

start on started ui
stop on stopping ui or starting halt or starting reboot

The messages log file typically shows debugd restarting right around the time when platform.CheckProcesses runs.

A fix here might be this test to first use "initctl list" to wait for there to not be any jobs in states other than start/running or stop/waiting. Here are some other states that I see when I run "restart ui":

ui start/killed, process 11430
arc-lifetime stop/post-stop, process 13994
ui start/post-stop, process 14005
machine-info start/pre-start, process 14038
ui start/starting
debugd start/pre-start, process 14181

Unfortunately, it looks like there's at least one small window where the ui job has started but debugd isn't yet in pre-start.

Maybe a better option would be to just make this test retry a few times until it sees all processes (although that may run the risk of not catching crash loops).
 

Comment 1 by derat@chromium.org, Jun 30 2018

Status: Started (was: Assigned)
Project Member

Comment 2 by bugdroid1@chromium.org, Jul 2

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/61ac86f0e649a3f1e87edacb7fb13d3511a81ee2

commit 61ac86f0e649a3f1e87edacb7fb13d3511a81ee2
Author: Daniel Erat <derat@chromium.org>
Date: Mon Jul 02 16:16:44 2018

tast-tests: Update platform.CheckProcesses to wait for jobs.

The example.DBus test restarts the ui Upstart job, which
triggers a restart of the debugd job. If
platform.CheckProcess runs soon afterward, the debugd
process often isn't running yet.

This change makes platform.CheckProcess wait for the debugd
(and also powerd, since it's had the same problem before)
Upstart jobs to be running before checking for processes.

BUG= chromium:859212 
TEST=ran example.DBus and platform.CheckProcesses one after
     another and checked that platform.CheckProcesses no
     longer reports that debugd isn't running

Change-Id: I8c3e87e9b0d5ddebfb30f68a3efbd49cae012650
Reviewed-on: https://chromium-review.googlesource.com/1121681
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Shuhei Takahashi <nya@chromium.org>

[modify] https://crrev.com/61ac86f0e649a3f1e87edacb7fb13d3511a81ee2/src/chromiumos/tast/local/bundles/cros/platform/check_processes.go

Status: Fixed (was: Started)

Sign in to add a comment