New issue
Advanced search Search tips

Issue 842453 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner:
Closed: May 2018
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 2
Type: Bug



Sign in to add a comment

Improve tast Autotest's handling of tast executable failures

Project Member Reported by derat@chromium.org, May 12 2018

Issue description

Right now, the tast.py Autotest server test doesn't do a great job of handling failed executions of the tast command.

If the initial "tast list" command succeeds (so we have a list of expected tests) but "tast run" fails (so we don't parse results.json to get the results of individual tests), tast.py doesn't write the individual tests to status.log.

Furthermore, the tast command only writes a results.json file after it's done running all tests. If it manages to run a partial set of tests but then dies for some reason, tast.py doesn't get any results. I should make tast write a file containing a stream of JSON result objects while it's reading control messages from test runners.
 
Project Member

Comment 1 by bugdroid1@chromium.org, May 15 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ece1bad856e9ef34b2e0d59f4dd2099fbb3d0090

commit ece1bad856e9ef34b2e0d59f4dd2099fbb3d0090
Author: Daniel Erat <derat@chromium.org>
Date: Tue May 15 18:28:03 2018

autotest: Improve handling of Tast execution failures.

Improve the "tast" server test's handling of failed
executions of the tast command:

Register the post-run hook earlier to ensure that not-run
tests will be reported in status.log even if tast fails
while running tests. (Note that the initial "tast list"
command still needs to succeed in order for us to get the
list of expected tests in the first place.)

Also include the final line of the command's stdout in the
first line of the TestFail exception make it easier to see
what went wrong.

BUG= chromium:770439 , chromium:831849 , chromium:842453 
TEST=modified the tast executable to fail before writing
     results.json and verified that status.log contains
     "Test was not run" FAIL results for all Tast tests and
     that the main "tast" result starts with "Failed to run
     tast (Failed to write results: intentional failure):
     Command ..."

Change-Id: I81bfa3eca80188fbe5c2e31316938fa2623a3156
Reviewed-on: https://chromium-review.googlesource.com/1056049
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Dan Erat <derat@chromium.org>

[modify] https://crrev.com/ece1bad856e9ef34b2e0d59f4dd2099fbb3d0090/server/site_tests/tast/tast.py

Project Member

Comment 2 by bugdroid1@chromium.org, May 18 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/tast/+/b5589dfd6993092cb0a04a481539c1acdf151551

commit b5589dfd6993092cb0a04a481539c1acdf151551
Author: Daniel Erat <derat@chromium.org>
Date: Fri May 18 02:35:03 2018

tast: Write streamed_results.jsonl.

Make the tast command write TestResults objects to a new
streamed_results.jsonl file while reading control messages
so we'll still have partial results if the tast process is
killed mid-run. The new file is in "JSON Lines" format, i.e.
newline-terminated JSON objects written by json.Encoder.
Started-but-not-completed tests will have end times of
0001-01-01T00:00:00Z in the new file.

BUG= chromium:842453 
TEST=added/updated unit tests; also updated resultsHandler
     to call os.Exit(1) abruptly and checked that
     streamed_results.jsonl contains full TestResult objects
     for completed tests and a partial object (i.e. with a
     zero-time 'end' property) for the in-progress test

Change-Id: I32e26986457bcca2a9d73ef4d87eeb94f3792e8a
Reviewed-on: https://chromium-review.googlesource.com/1063177
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Dan Erat <derat@chromium.org>

[modify] https://crrev.com/b5589dfd6993092cb0a04a481539c1acdf151551/src/chromiumos/cmd/tast/run/results_test.go
[modify] https://crrev.com/b5589dfd6993092cb0a04a481539c1acdf151551/src/chromiumos/cmd/tast/run/results.go

Project Member

Comment 3 by bugdroid1@chromium.org, May 22 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/026f07476817332572fee92677f1f3291627905d

commit 026f07476817332572fee92677f1f3291627905d
Author: Daniel Erat <derat@chromium.org>
Date: Tue May 22 04:17:56 2018

autotest: Update tast server test to read streamed results.

Update the tast server test to read test results from
streamed_results.jsonl rather than results.json. The tast
executable writes results to streamed_results.jsonl as
they're received, so this allows partial results to be
reported by Autotest even if the tast process is killed
before it finishes.

BUG= chromium:842453 
TEST=updated tast to die after writing results for a few
     tests and checked status.log to confirm that a) the
     completed tests are still reported as passing, b) the
     test that was in progress when the tast process exited
     gets a 'Test did not finish' failure, and c) the
     not-run tests get 'Test was not run' failures

Change-Id: I72b08ec0bbaf83328a27c60d7d6a7156eebf397d
Reviewed-on: https://chromium-review.googlesource.com/1065254
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>

[modify] https://crrev.com/026f07476817332572fee92677f1f3291627905d/server/site_tests/tast/tast.py

Comment 4 by derat@chromium.org, May 25 2018

Status: Fixed (was: Started)

Sign in to add a comment