Factory: pytest shutdown has race condition |
|||
Issue description
When running pytest shutdown, this has race condition.
1. invocation:
invocation need to run pytest (call goofy_rpc.Shutdown) and record logs.
2. goofy:
When goofy_rpc.Shutdown() is called, goofy call destroy().
if 1. run first, this can finish normally.
if 2. run first, goofy close event_log and testlog, then 1. can't record the end event logs.
Therefore the logs will be:
factory.log:
STARTING -> PASSED -> reboot -> STARTING -> PASSED
event_log and Testlog:
STARTING (-> PASSED) -> reboot -> STARTING -> PASSED
,
Jun 20 2017
Can post the extra exception that will be printed for scenario (2) ?
,
Jun 20 2017
[ERROR] goofy invocation.py:693 2017-06-19 17:29:38.426 Unable to post DESTROY_TEST event
Traceback (most recent call last):
File "/usr/local/factory/py_pkg/cros/factory/goofy/invocation.py", line 688, in _run
self.goofy.event_client.post_event(
AttributeError: 'NoneType' object has no attribute 'post_event'
[ERROR] goofy testlog.py:354 2017-06-19 17:29:38.447 Not able to collect /var/factory/log/running/dff7f429-bf0f-493d-a4df-599aefe28b45-session.json. Last read: {---SKIP---}
Traceback (most recent call last):
File "/usr/local/factory/py_pkg/cros/factory/testlog/testlog.py", line 350, in LogTestRun
Log(test_run)
File "/usr/local/factory/py_pkg/cros/factory/testlog/testlog.py", line 398, in Log
testlog_singleton = GetGlobalTestlog()
File "/usr/local/factory/py_pkg/cros/factory/testlog/testlog.py", line 369, in GetGlobalTestlog
Testlog()
File "/usr/local/factory/py_pkg/cros/factory/testlog/testlog.py", line 159, in __init__
session_data = Testlog._ReadSessionInfo()
File "/usr/local/factory/py_pkg/cros/factory/testlog/testlog.py", line 230, in _ReadSessionInfo
'Not able to find environment variable %r' % TESTLOG_ENV_VARIABLE_NAME)
AssertionError: Not able to find environment variable 'TESTLOG'
These two are the exception logs from event_client and testlog when I skip event_log.Log().
If you don't skip event_log.Log(), you'll get "Unable to log end_test event" and won't get the exception from testlog.
,
Jun 22 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/factory/+/9d675c6e3672d4c7b711f3260a2a0b22c9b920ba commit 9d675c6e3672d4c7b711f3260a2a0b22c9b920ba Author: chuntsen <chuntsen@google.com> Date: Thu Jun 22 14:53:34 2017 goofy: Join invocation threads in the beginning of destroy invocation.py can't access event_client, event_log and testlogs after goofy.destory(). Otherwise, some logs will be missing with confusing exception raised. We should join invocation threads to avoid race condition between invocation.py and goofy.destroy(). Also fix wrong logging messages. BUG= chromium:734533 TEST=manually test on local device, make test Change-Id: I3474cde7acf5a0b8b7c5ab03d0f4a6ec2a823bff Reviewed-on: https://chromium-review.googlesource.com/541098 Commit-Ready: Chun-Tsen Kuo <chuntsen@chromium.org> Tested-by: Chun-Tsen Kuo <chuntsen@chromium.org> Reviewed-by: Hung-Te Lin <hungte@chromium.org> [modify] https://crrev.com/9d675c6e3672d4c7b711f3260a2a0b22c9b920ba/py/goofy/goofy.py
,
Jun 23 2017
,
Jan 22 2018
|
|||
►
Sign in to add a comment |
|||
Comment 1 by chuntsen@chromium.org
, Jun 20 2017