cyan-paladin:6819 failed: cheets_Midis: midis didn't restart. |
||||||
Issue descriptionFiled by sheriff-o-matic@appspot.gserviceaccount.com on behalf of tfiga@google.com cyan-paladin:6819 failed Builders failed on: - cyan-paladin: https://cros-goldeneye.corp.google.com/chromeos/healthmonitoring/buildDetails?buildbucketId=8937645124517358896 6818 and 6820 failed too: https://luci-milo.appspot.com/buildbot/chromeos/cyan-paladin/6818 https://luci-milo.appspot.com/buildbot/chromeos/cyan-paladin/6820 08/20 21:01:16.975 DEBUG| utils:0218| Running 'pgrep midis' 08/20 21:01:16.976 DEBUG| global_hooks:0056| 'pgrep midis' 08/20 21:01:17.045 DEBUG| test:0381| Test failed due to midis didn't restart.. Exception log follows the after_iteration_hooks. 08/20 21:01:17.046 DEBUG| test:0386| Starting after_iteration_hooks for cheets_Midis 08/20 21:01:17.053 DEBUG| base_sysinfo:0124| Loggable saves logs to /usr/local/autotest/results/default/cheets_Midis/sysinfo/iteration.1/interrupts.after 08/20 21:01:17.055 DEBUG| utils:0218| Running 'mkdir -p /usr/local/autotest/results/default/cheets_Midis/sysinfo/iteration.1/var/spool' 08/20 21:01:17.061 DEBUG| global_hooks:0056| 'mkdir -p /usr/local/autotest/results/default/cheets_Midis/sysinfo/iteration.1/var/spool' 08/20 21:01:17.119 DEBUG| utils:0218| Running 'rsync --no-perms --chmod=ugo+r -a --safe-links --exclude=/crash/**autoserv* --exclude=/crash/*.core /var/spool/crash /usr/local/autotest/results/default/cheets_Midis/sysinfo/iteration.1/var/spool' 08/20 21:01:17.120 DEBUG| global_hooks:0056| 'rsync --no-perms --chmod=ugo+r -a --safe-links --exclude=/crash/**autoserv* --exclude=/crash/*.core /var/spool/crash /usr/local/autotest/results/default/cheets_Midis/sysinfo/iteration.1/var/spool' 08/20 21:01:17.163 DEBUG| utils:0218| Running 'rm -rf /var/spool/crash/*' 08/20 21:01:17.164 DEBUG| global_hooks:0056| 'rm -rf /var/spool/crash/*' 08/20 21:01:17.203 DEBUG| base_sysinfo:0124| Loggable saves logs to /usr/local/autotest/results/default/cheets_Midis/sysinfo/iteration.1/meminfo.after 08/20 21:01:17.210 DEBUG| base_sysinfo:0124| Loggable saves logs to /usr/local/autotest/results/default/cheets_Midis/sysinfo/iteration.1/slabinfo.after 08/20 21:01:17.212 DEBUG| base_sysinfo:0124| Loggable saves logs to /usr/local/autotest/results/default/cheets_Midis/sysinfo/iteration.1/schedstat.after 08/20 21:01:17.214 DEBUG| utils:0218| Running 'logger "autotest finished iteration /usr/local/autotest/results/default/cheets_Midis/sysinfo/iteration.1"' 08/20 21:01:17.218 DEBUG| global_hooks:0056| 'logger "autotest finished iteration /usr/local/autotest/results/default/cheets_Midis/sysinfo/iteration.1"' 08/20 21:01:17.264 DEBUG| test:0391| after_iteration_hooks completed 08/20 21:01:17.266 WARNI| test:0606| The test failed with the following exception Traceback (most recent call last): File "/usr/local/autotest/common_lib/test.py", line 600, in _exec _call_test_function(self.execute, *p_args, **p_dargs) File "/usr/local/autotest/common_lib/test.py", line 800, in _call_test_function return func(*args, **dargs) File "/usr/local/autotest/common_lib/test.py", line 464, in execute postprocess_profiled_run, args, dargs) File "/usr/local/autotest/common_lib/test.py", line 371, in _call_run_once self.run_once(*args, **dargs) File "/usr/local/autotest/tests/cheets_Midis/cheets_Midis.py", line 70, in run_once raise error.TestFail('midis didn\'t restart.') TestFail: midis didn't restart.
,
Aug 21
,
Aug 21
That's odd. Should be able to restart. Can you point me to where I can look at the past results of this autotest? No one has checked anything into the midis/ folder recently, so I'd like to identify when this started failing.
,
Aug 22
Prashant, thanks for taking a look. You can check previous runs on this board as follows: 1) Open builder history: https://luci-milo.appspot.com/buildbot/chromeos/cyan-paladin/ 2) Click any build number you want to check, e.g. #6819 as in this report. The shortcut there is https://luci-milo.appspot.com/buildbot/chromeos/cyan-paladin/6819 , where you can just change board names and build numbers as you want. 3) Find the failure in HWTest section and click the "[Test-Logs]: cheets_Midis: retry_count: 2, FAIL: midis didn't restart." (or similar) link. This should open Stainless log viewer with all logs from the failed test run on the DUT.
,
Aug 22
+other sheriffs too
,
Aug 22
cheets_Midis has seemed very stable across all platforms off late. I wonder if this is just a cyan-specific issue (midis not being closed, so the PID just refers to the old process which is still running?)? Can I borrow a cyan from you? If so, I can try and debug this, but Chrome should be shutting down midis when it itself logs out, so this might be some signal not working on cyan: "stop on stop-arc-instance" is where it should be getting killed, so maybe stop-arc-instance isn't getting emitted any longer (on cyan only?) ?
,
Aug 22
Same problem happened today on auron_yuna: https://luci-milo.appspot.com/buildbot/chromeos/auron_yuna-paladin/3729
,
Aug 22
CC-ing cmtm@ Hi Chris, It seems like the midis daemon is not restarting after I shut down a Chrome session in autotest and get to the login screen again,rather, the previous instance continues running and doesn't seem to get killed on stop-arc-instance. This didn't seem to be the behavior before (midis would get killed on logging out, and then restart). Has something changed in the way the upstart signals are emitted? If not, would you happen to know why the stop-arc-instance signal is being ignored? Relevant error line is located here: https://cs.corp.google.com/android/external/autotest/client/site_tests/cheets_Midis/cheets_Midis.py?rcl=043e2ad35875a92065f50f3c01ae1261b647d467&l=69 Thanks!
,
Aug 23
#6, I guess you could borrow one from me, but it may take a while to get it from TOK to MTV. ;) I think you should be able to find one either in your office or just use one in the lab in pool:suites (go/chromeos-autotest-faq).
,
Aug 23
+derat@, +apronin@, possibly this is something related to the dbus problems in session manager ( crbug.com/876196 , crbug.com/876197 )?
,
Aug 23
#10: Not sure; yusukes@ might know more about the expected lifetimes for ARC-related processes. Maybe the midis Upstart job should also stop if the 'ui' job is stopped.
,
Aug 28
This seems to no longer be an issue (i.e the underlying ARC++ startup/crash issue (judging by the state of cyan-paladin) |
||||||
►
Sign in to add a comment |
||||||
Comment 1 by tfiga@chromium.org
, Aug 21Owner: pmalani@chromium.org