In the tricky-release run at http://stainless/browse/chromeos-autotest-results/235396368-chromeos-test/, the ui.SessionManagerRespawn test failed while trying to restart the ui job at completion:

2018/09/07 23:36:23 Started test ui.SessionManagerRespawn
2018/09/07 23:36:23 [23:36:22.567] Getting initial session_manager process
2018/09/07 23:36:23 [23:36:22.568] Initial session_manager process is 7871
2018/09/07 23:36:23 [23:36:22.568] Killing 7871
2018/09/07 23:36:23 [23:36:22.568] Waiting for session_manager to respawn
2018/09/07 23:36:24 [23:36:24.223] Respawned session_manager process is 8671
2018/09/07 23:36:24 [23:36:24.223] Repeatedly killing session_manager to check that ui-respawn stops restarting it eventually
2018/09/07 23:36:24 [23:36:24.223] Killing 8671 and watching for respawn
2018/09/07 23:36:38 [23:36:37.770] session_manager (correctly) not respawned
2018/09/07 23:36:38 [23:36:37.778] ui job not running; starting it
2018/09/07 23:36:38 [23:36:37.785] External command failed: exit status 1
2018/09/07 23:36:38 [23:36:37.785] Command: 'initctl' 'start' 'ui'
2018/09/07 23:36:38 [23:36:37.785] Uncaptured output:
initctl: Job is already running: ui
2018/09/07 23:36:38 [23:36:37.785] Error at session_manager_respawn.go:74: Failed to ensure ui job is running: restarting job "ui" failed: exit status 1

Here's the end of the 'messages' log:

2018-09-07T23:36:24.206218-07:00 INFO session_manager[8671]: [INFO:session_manager_service.cc(159)] SessionManagerService starting
2018-09-07T23:36:24.210664-07:00 INFO session_manager[8671]: [INFO:browser_job.cc(148)] Running browser /opt/google/chrome/chrome ...
2018-09-07T23:36:24.224306-07:00 WARNING kernel: [ 2460.165363] init: ui main process (8671) killed by KILL signal
2018-09-07T23:36:24.318124-07:00 NOTICE ui-respawn[8744]: ui died on signal KILL.
2018-09-07T23:36:24.361973-07:00 NOTICE ui-respawn[8798]: Respawning ui.
2018-09-07T23:36:37.864443-07:00 CRIT ui-post-stop-unkillable[8936]:  8683 Z<sl [chrome] <defunct>
2018-09-07T23:36:37.864462-07:00 CRIT ui-post-stop-unkillable[8936]:  8783 S    [chrome]
2018-09-07T23:36:37.864470-07:00 CRIT ui-post-stop-unkillable[8936]:  8822 D    [nacl_helper]
2018-09-07T23:36:37.864650-07:00 CRIT ui-post-stop-unkillable[8936]:  8825 D    [nacl_helper_non]
2018-09-07T23:36:37.864694-07:00 CRIT ui-post-stop-unkillable[8936]:  8856 Z<   [chrome] <defunct>
2018-09-07T23:36:37.864730-07:00 CRIT ui-post-stop-unkillable[8936]:  8899 Zl   [chrome] <defunct>
2018-09-07T23:36:37.864766-07:00 CRIT ui-post-stop-unkillable[8936]:  8908 Zl   [chrome] <defunct>
2018-09-07T23:36:37.864800-07:00 CRIT ui-post-stop-unkillable[8936]:  8911 Zl   [chrome] <defunct>

I see a ui-post-stop-unkillable message in issue 625159, although I think that my conclusion there that session_manager was hanging during keygen was probably incorrect (due to confusing logging in session_manager at the time).

I should probably make this test use faillog, as there's a chance that seeing a full process list would've provided more clues about what went wrong.

I should also make upstart.EnsureJobRunning log more information about the job's state when RestartJob fails. I suspect that the job was actually stuck in the "stopping" state when we tried to start it -- we should possibly wait a bit here.