Tast should let tests report key-value pairs |
||||
Issue descriptionTast should let tests report key-value pairs containing performance (or other) data. Right now, tests can write arbitrary output files that get copied to the Tast results dir, but we should have something that ties in with the existing infrastructure for collecting and viewing these values. I'm not familiar with the Autotest implementation (I've never used it), but there appears to be documentation for an old system at https://www.chromium.org/chromium-os/testing/autotest-keyvals/perf-keyvals and a new system at http://www.chromium.org/chromium-os/testing/perf-data. I'm not sure which we use. I see many references to both write_perf_keyval() and output_perf_value() in tests in the Autotest repo, so... both, I guess? It looks like the resulting values are parsed by the TKO parser. I'd guess that something like this might work for Tast: a) testing.State can expose a new method for tests to use to record values. b) control.TestEnd can include a map of values (to pass them from test bundles to the tast process). c) When cmd/tast/run/results.go processes TestEnd messages, it can collect the results and then write them to a file in the results dir. d) The tast.py Autotest code can then read this file (similar to how it reads test results from streamed_results.jsonl) and report the values for the TKO parser. The existing output_perf_value() function in client/common_lib/test.py looks like it writes to a results-chart.json file. This is visible for the enterprise_CFM_Perf test run at http://stainless/browse/chromeos-autotest-results/221267594-chromeos-test/ , for example. I suspect that there will be challenges around making the parser support reading values from multiple Tast tests in a single Autotest results dir. nya@, feel free to assign this to me if you don't want to work on it. :-)
,
Jul 29
The code that reads results-chart.json[1] probably isn't going to know how to interpret results from multiple Tast tests if they appear in a single file, though, right? 1. I see multiple files that appear to parse it: src/third_party/autotest/files/tko/models.py src/third_party/autotest/files/client/common_lib/test.py src/third_party/toolchain-utils/crosperf/results_cache.py etc.
,
Jul 30
I'm not sure the how tied tast related to the existing autotest infrastructure. The file output by the output_perf_value is picked up by the post-test function located in tko/parse.py & tko/parsers/version_1.py mostly and sent to chromeperf in tko/perf_upload/perf_uploader.py. You might want to make sure the existing parser parsed the tast result correctly. Or implement your new parser and sent the information to chromeperf by adding a new parser to scheduler/postjob_task.py
,
Jul 31
Thanks for pointers! I think I can make a small change to TKO parser to make it work for Tast. I assume perf reporting via keyval is deprecated, so for now I'll implement support of results-chart.json only.
,
Aug 2
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/platform/tast-tests/+/da505e001ac3ce0dc38bac7b6faadbf25c885de4 commit da505e001ac3ce0dc38bac7b6faadbf25c885de4 Author: Shuhei Takahashi <nya@chromium.org> Date: Thu Aug 02 20:53:31 2018 perf: Support recording performance metrics. BUG= chromium:868593 TEST=fast_build.sh -T TEST=tast run DUT example.Perf Change-Id: I8a3bb4371a017e8d63e01fcf4ba4adcdfa7b35d3 Reviewed-on: https://chromium-review.googlesource.com/1157743 Commit-Ready: Shuhei Takahashi <nya@chromium.org> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Shuhei Takahashi <nya@chromium.org> [add] https://crrev.com/da505e001ac3ce0dc38bac7b6faadbf25c885de4/src/chromiumos/tast/local/perf/testdata/TestSetMultiple.json [add] https://crrev.com/da505e001ac3ce0dc38bac7b6faadbf25c885de4/src/chromiumos/tast/local/perf/testdata/TestAppendMultiple.json [add] https://crrev.com/da505e001ac3ce0dc38bac7b6faadbf25c885de4/src/chromiumos/tast/local/perf/perf.go [add] https://crrev.com/da505e001ac3ce0dc38bac7b6faadbf25c885de4/src/chromiumos/tast/local/bundles/cros/example/perf.go [add] https://crrev.com/da505e001ac3ce0dc38bac7b6faadbf25c885de4/src/chromiumos/tast/local/perf/perf_test.go [add] https://crrev.com/da505e001ac3ce0dc38bac7b6faadbf25c885de4/src/chromiumos/tast/local/perf/testdata/TestSave.json [add] https://crrev.com/da505e001ac3ce0dc38bac7b6faadbf25c885de4/src/chromiumos/tast/local/perf/testdata/TestSetSingle.json
,
Aug 7
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/0ae6c4d45e07db7266a9a99f929737c1625fd969 commit 0ae6c4d45e07db7266a9a99f929737c1625fd969 Author: Shuhei Takahashi <nya@chromium.org> Date: Tue Aug 07 08:51:05 2018 tko: Support reporting perf values from Tast tests. Autotest tests and Tast tests have different directory structures, so this CL updates TKO parser to read perf values from different location for Tast tests. I also considered setting `subdir` to Tast test results so that we can reuse Autotest code path (just below the code part this CL make changes), but I did not so because (1) nesting "results" dir like tast/results/tests/example.Pass/results/results-chart.json is verbose, and (2) setting `subdir` has side effects affecting other parts of TKO parser. BUG= chromium:868593 TEST=tko/parse --dry-run -o /path/to/results # w/ mods to parse.py Change-Id: I2564b47febe2c43cd2a0fb0d868068a820b4747e Reviewed-on: https://chromium-review.googlesource.com/1156194 Commit-Ready: ChromeOS CL Exonerator Bot <chromiumos-cl-exonerator@appspot.gserviceaccount.com> Tested-by: Shuhei Takahashi <nya@chromium.org> Reviewed-by: Dan Erat <derat@chromium.org> [modify] https://crrev.com/0ae6c4d45e07db7266a9a99f929737c1625fd969/tko/models.py [add] https://crrev.com/0ae6c4d45e07db7266a9a99f929737c1625fd969/tko/tast.py
,
Sep 25
|
||||
►
Sign in to add a comment |
||||
Comment 1 by ihf@google.com
, Jul 28