Sounds good. Reusing the parse.py allows the integration to other results dashboard, e.g., Autotest tko, Sponge (GoldenEye imports Sponge data), wmatrix, and the big query based dashboard nya@ is building.

Even if we end up tossing a lot of what's in parse.py , reusing the status.log format sounds like a good idea. It's among the closest things autotest has to a well defined API.

After some discussion with Aviv and helpful pointers from Allen, it sounds like the best approach right now is probably to make the tast_Runner Autotest server test add additional status.log entries describing the results of individual Tast tests. Here's what Allen wrote over email:

"The code that does this for Autotest is:

* run_test() entry point: https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/client/bin/job.py?l=550
* START/END for test (group): https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/client/bin/job.py?l=620
* GOOD/ERROR for test: https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/client/bin/job.py?l=541

Luckily, test instances have a reference to job, so you can just call (https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/client/site_tests/dummy_Fail/dummy_Fail.py?l=35) record on the job from tast_Runner.  You can use the above examples to conform to how Autotest emits entries for individual Autotest tests and do the same for tast tests."

I'm testing a tast_Runner change to do this now.

Feature request: Allow tests to send an email when they fail. (Maybe open a new bug for that?)

I'm using the tast test ui.MashLogin for an experimental UI feature and just need passive updates for failures.

#9: That's a bit out of scope for this bug, but I've sent email to http://g/chromeos-infra-discuss/llBGUZ7kyZ0/discussion asking for advice about it.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/aecc41f82f83d9481503c0f2537f386cdb9b7e17

commit aecc41f82f83d9481503c0f2537f386cdb9b7e17
Author: Daniel Erat <derat@chromium.org>
Date: Fri Mar 23 13:18:02 2018

autotest: Make tast_Runner write individual test results.

Update the tast_Runner server test to include each
Tast-based test's results in its Autotest TKO status.log
file. This is off by default since tast_Runner doesn't
currently run under SSP, giving no way to check that that
this works in production until this change is pushed to
prod.

Also add a control.example file to make it easier to run
all example Tast tests.

BUG= chromium:770439 
TEST=ran using test_that and verified that status.log
     includes results from individual tast tests

Change-Id: I207fa1a6acbb6fd958b2508b561d9fb5569711de
Reviewed-on: https://chromium-review.googlesource.com/974643
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Dan Erat <derat@chromium.org>

[add] https://crrev.com/aecc41f82f83d9481503c0f2537f386cdb9b7e17/server/site_tests/tast_Runner/control.example
[modify] https://crrev.com/aecc41f82f83d9481503c0f2537f386cdb9b7e17/server/site_tests/tast_Runner/tast_Runner.py

The code in #11 doesn't work because dateutil.parser apparently isn't available on shards. Guess it's time to write a regexp to parse RFC 3339 dates. :-/

#11 also broke tast_Runner on release builders. :-( Sent a fix for review: https://crrev.com/c/979124

Allen, I ran a couple of tryjobs with the updated tast_Runner code to report the results of individual Tast tests via the status.log file, but I'm not sure that the Tast results are being extracted. For example, http://cautotest-prod/afe/#tab_id=view_job&object_id=186169035 is a tast_Runner job that ran on kevin-release-tryjob. Its status.log file is at https://storage.cloud.google.com/chromeos-autotest-results/186169035-chromeos-test/chromeos6-row4-rack16-host4/status.log and looks like this:

START	tast_Runner	tast_Runner	timestamp=1521881219	localtime=Mar 24 01:46:59	
	START	----	----	timestamp=1521881240	localtime=Mar 24 01:47:20	
		GOOD	----	sysinfo.before	timestamp=1521881240	localtime=Mar 24 01:47:20	
	END GOOD	----	----	timestamp=1521881240	localtime=Mar 24 01:47:20	
	START	----	----	timestamp=1521881251	localtime=Mar 24 01:47:31	
		GOOD	----	sysinfo.iteration.before	timestamp=1521881252	localtime=Mar 24 01:47:32	
	END GOOD	----	----	timestamp=1521881252	localtime=Mar 24 01:47:32	
	START	----	power.CheckStatus	timestamp=1521881256	localtime=Mar 24 01:47:36	
		GOOD	----	power.CheckStatus	timestamp=1521881257	localtime=Mar 24 01:47:37	
	END GOOD	----	power.CheckStatus	timestamp=1521881257	localtime=Mar 24 01:47:37	
...	
	START	----	power.Reboot	timestamp=1521881295	localtime=Mar 24 01:48:15	
		GOOD	----	power.Reboot	timestamp=1521881311	localtime=Mar 24 01:48:31	
	END GOOD	----	power.Reboot	timestamp=1521881311	localtime=Mar 24 01:48:31	
	START	----	----	timestamp=1521881332	localtime=Mar 24 01:48:52	
		GOOD	----	sysinfo.iteration.after	timestamp=1521881332	localtime=Mar 24 01:48:52	
	END GOOD	----	----	timestamp=1521881332	localtime=Mar 24 01:48:52	
	START	----	----	timestamp=1521881343	localtime=Mar 24 01:49:03	
		GOOD	----	sysinfo.after	timestamp=1521881343	localtime=Mar 24 01:49:03	
	END GOOD	----	----	timestamp=1521881343	localtime=Mar 24 01:49:03	
	GOOD	tast_Runner	tast_Runner	timestamp=1521881352	localtime=Mar 24 01:49:12	completed successfully
END GOOD	tast_Runner	tast_Runner	timestamp=1521881352	localtime=Mar 24 01:49:12	
...

When I search for "power.Reboot" using the cautotest job page or Stainless, I don't get any results. Where should the Tast tests be visible?

You can check job_reporter_output.log for the tko parse logs.

I took a look at the parsing code.  I haven't read the code line by line yet, but it looks like we ignore nested tests (sorry about the misinformation previously).

https://cs.corp.google.com/chromeos_public/src/third_party/autotest/files/tko/parsers/version_1.py?l=405

You'll need to somehow write the tests without nesting them.  I can't think of an obvious elegant solution at the moment.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/c65fa4701d560257375951009cc60d5b1a1b9c11

commit c65fa4701d560257375951009cc60d5b1a1b9c11
Author: Daniel Erat <derat@chromium.org>
Date: Tue Mar 27 03:16:47 2018

project-moblab: Install dev-python/python-dateutil.

Make the chromeos-bsp-moblab package depend on
dev-python/python-dateutil. The dateutil module is used by
the tast_Runner server test.

BUG= chromium:770439 
TEST=none

Change-Id: I303d4da296e899858361cf382c3d0a8fe36d613d
Reviewed-on: https://chromium-review.googlesource.com/981316
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Jason Clinton <jclinton@chromium.org>

[modify] https://crrev.com/c65fa4701d560257375951009cc60d5b1a1b9c11/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-9999.ebuild

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/39b7b782850544619ada176cb8ceb7d60d61518c

commit 39b7b782850544619ada176cb8ceb7d60d61518c
Author: Daniel Erat <derat@chromium.org>
Date: Tue Mar 27 03:16:45 2018

autotest: Enable SSP for the tast_Runner test.

Enable server-side packaging for the tast_Runner server
test. While the tast executable and server test bundle are
still installed via prod pushes rather than SSP, this at
least makes it possible to test changes to tast_Runner via
tryjobs and makes it possible to deploy changes to it
without waiting for prod pushes.

BUG= chromium:770439 
TEST=none

Change-Id: I6f2a40646e8d9a0e038d2b298d3cbc9501d657a5
Reviewed-on: https://chromium-review.googlesource.com/979336
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Dan Erat <derat@chromium.org>

[modify] https://crrev.com/39b7b782850544619ada176cb8ceb7d60d61518c/server/site_tests/tast_Runner/control.example
[modify] https://crrev.com/39b7b782850544619ada176cb8ceb7d60d61518c/server/site_tests/tast_Runner/control.bvt_arc
[modify] https://crrev.com/39b7b782850544619ada176cb8ceb7d60d61518c/server/site_tests/tast_Runner/control.bvt

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/a749a794a0b940b34fca9f24bba4070f6d076c6d

commit a749a794a0b940b34fca9f24bba4070f6d076c6d
Author: Daniel Erat <derat@chromium.org>
Date: Tue Mar 27 03:16:41 2018

autotest: Add python-dateutil to external_packages.py.

Add the python-dateutil package
(https://dateutil.readthedocs.io/en/stable/), needed by the
tast_Runner server test to parse RFC3339 dates.

BUG= chromium:770439 
TEST=ran build_externals.py

Change-Id: Ieb0a76ff381b6c6f0798a22cfc88e0daefbad9f8
Reviewed-on: https://chromium-review.googlesource.com/981179
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>

[modify] https://crrev.com/a749a794a0b940b34fca9f24bba4070f6d076c6d/utils/external_packages.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/8184ad2089e5283b20024c895a86977209511c44

commit 8184ad2089e5283b20024c895a86977209511c44
Author: Daniel Erat <derat@chromium.org>
Date: Mon Apr 09 22:18:39 2018

autotest: Write test results for tast_Runner.

Add a server_job.add_post_run_hook method and make
tast_Runner use it to write results for individual Tast
tests as top-level results in the status.log file.

BUG= chromium:770439 
TEST=ran tast_Runner and verified that status.log file is as
     expected

Change-Id: I3ee14934b9e82779c394cc261df86cf8de078d81
Reviewed-on: https://chromium-review.googlesource.com/997014
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Allen Li <ayatane@chromium.org>

[modify] https://crrev.com/8184ad2089e5283b20024c895a86977209511c44/server/server_job.py
[modify] https://crrev.com/8184ad2089e5283b20024c895a86977209511c44/server/site_tests/tast_Runner/control.example
[modify] https://crrev.com/8184ad2089e5283b20024c895a86977209511c44/server/site_tests/tast_Runner/tast_Runner.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/5b686004294ed73d6e0f25e6e39d2303eb3a6117

commit 5b686004294ed73d6e0f25e6e39d2303eb3a6117
Author: Daniel Erat <derat@chromium.org>
Date: Wed Apr 18 09:07:49 2018

autotest-server-tests-tast: Rename "tast_Runner" to "tast".

Rename the "tast_Runner" Autotest server test to "tast" in
autotest-server-tests-tast.

Also switch to EAPI 5.

BUG= chromium:770439 
TEST=none
CQ-DEPEND=I61fb3e4612d2f2386fffb4387a2a5172369481f1

Change-Id: I872d2b2285d926693db0fe4a7f26f43e55a7a86a
Reviewed-on: https://chromium-review.googlesource.com/1015775
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>

[modify] https://crrev.com/5b686004294ed73d6e0f25e6e39d2303eb3a6117/chromeos-base/autotest-server-tests-tast/autotest-server-tests-tast-9999.ebuild

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/5d7e3788392cea52e3b0c3281c02ebad2cd9719e

commit 5d7e3788392cea52e3b0c3281c02ebad2cd9719e
Author: Daniel Erat <derat@chromium.org>
Date: Wed Apr 18 09:07:48 2018

autotest: Rename tast_Runner to "tast".

Rename the tast_Runner server test to "tast". This will
result in friendlier result names for individual Tast-based
tests (e.g. "tast.power.CheckStatus").

BUG= chromium:770439 
TEST=ran with test_that
CQ-DEPEND=I872d2b2285d926693db0fe4a7f26f43e55a7a86a

Change-Id: I61fb3e4612d2f2386fffb4387a2a5172369481f1
Reviewed-on: https://chromium-review.googlesource.com/1004009
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Ilja H. Friedel <ihf@chromium.org>

[rename] https://crrev.com/5d7e3788392cea52e3b0c3281c02ebad2cd9719e/server/site_tests/tast/control.example
[rename] https://crrev.com/5d7e3788392cea52e3b0c3281c02ebad2cd9719e/server/site_tests/tast/control.bvt
[rename] https://crrev.com/5d7e3788392cea52e3b0c3281c02ebad2cd9719e/server/site_tests/tast/control.bvt_arc
[rename] https://crrev.com/5d7e3788392cea52e3b0c3281c02ebad2cd9719e/server/site_tests/tast/tast.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ece1bad856e9ef34b2e0d59f4dd2099fbb3d0090

commit ece1bad856e9ef34b2e0d59f4dd2099fbb3d0090
Author: Daniel Erat <derat@chromium.org>
Date: Tue May 15 18:28:03 2018

autotest: Improve handling of Tast execution failures.

Improve the "tast" server test's handling of failed
executions of the tast command:

Register the post-run hook earlier to ensure that not-run
tests will be reported in status.log even if tast fails
while running tests. (Note that the initial "tast list"
command still needs to succeed in order for us to get the
list of expected tests in the first place.)

Also include the final line of the command's stdout in the
first line of the TestFail exception make it easier to see
what went wrong.

BUG= chromium:770439 , chromium:831849 , chromium:842453 
TEST=modified the tast executable to fail before writing
     results.json and verified that status.log contains
     "Test was not run" FAIL results for all Tast tests and
     that the main "tast" result starts with "Failed to run
     tast (Failed to write results: intentional failure):
     Command ..."

Change-Id: I81bfa3eca80188fbe5c2e31316938fa2623a3156
Reviewed-on: https://chromium-review.googlesource.com/1056049
Commit-Ready: Dan Erat <derat@chromium.org>
Tested-by: Dan Erat <derat@chromium.org>
Reviewed-by: Dan Erat <derat@chromium.org>

[modify] https://crrev.com/ece1bad856e9ef34b2e0d59f4dd2099fbb3d0090/server/site_tests/tast/tast.py