Currently, tast tests don't fit in too well with chrome's test results reporting & recording. One run of chrome_all_tast_tests is considered just a single test, with only a pass or fail result. But we've got the ability to track a lot more than that (individual tests within a suite, run duration, output snippets). It's essentially everything outlined in:
https://chromium.googlesource.com/chromium/src/+/master/docs/testing/json_test_results_format.md
eg: See https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/chromeos-amd64-generic-rel/24394 where video.WebRTCCamera and video.WebRTCPeerConnCameraVP8 failed, but the build page just says "chrome_all_tast_tests (status FAILURE)".
Compare that with cc_unittests in https://luci-milo.appspot.com/p/chromium/builders/luci.chromium.try/chromeos-amd64-generic-rel/171724. The suite failed 6 tests cases, each of which are printed in the build page and also have a link to a log of the test's output (https://logs.chromium.org/logs/chromium/buildbucket/cr-buildbucket.appspot.com/8924022334701275504/+/steps/cc_unittests__with_patch_/0/logs/KeyframeModelTest.TrimTimeAlternateReverseTwoIterationsPlaybackNormalAlternate__status_FAILURE_/0)
It would be great if we could get tast tests to utilize the same result reporting. Right now that's a bit difficult since we're using local_test_runner on the device itself, so I think the only option there is to parse the stdout, which I'm not too wild about.
The tast binary is more feature-rich FWIU, so switching to that should make hooking up the results a lot easier. The only problem is getting access to the tool on chrome bots, which don't have a cros chroot checkout. One option is to package the tool into the simplechrome sdk we download (not sure how easy that would be). Another option is to package it up in CIPD, which our bots know well. It looks it was being CIPD'ed at one point, but stopped in https://chromium-review.googlesource.com/c/chromiumos/chromite/+/1086160/. Not sure why it was reverted.. maybe we could add it back?
Tentatively assigning to dan so we can start discussing this when he's back from OOO
Comment 1 by derat@chromium.org
, Jan 20 (2 days ago)Owner: bpastene@chromium.org