To support new test suites such as webkit layout tests, add a parser for json_test_results_format so that Findit can understand the test results.
chanli@, do you have an update on this? (It came up in triage of P2 issues older than 60 days.)
I'm currently working on it(forgot to change bug status). I will update this bug when my changes are ready.
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/2fb7d5bb21416bbbd3619890b74da67430c5e413 commit 2fb7d5bb21416bbbd3619890b74da67430c5e413 Author: Chan <chanli@chromium.org> Date: Tue May 29 18:38:07 2018 [Findit] Classify gtest results in a new format. To be compatible with json test results, classify gtest results in a new format. The new method will be used to pre-process test results of test reruns (both deflake and determine pass rate), and then the result will be proccessed differently based on the purpose of the swarming task(This part of changes will be in a following CL). Bug: 806002 Change-Id: I07e588285b9ffec7d57602eadb154a8c0fb3ba12 Reviewed-on: https://chromium-review.googlesource.com/1067885 Commit-Queue: Chan Li <chanli@chromium.org> Reviewed-by: Shuotao Gao <stgao@chromium.org> [add] https://crrev.com/2fb7d5bb21416bbbd3619890b74da67430c5e413/appengine/findit/libs/test_results/test/classified_test_results_test.py [modify] https://crrev.com/2fb7d5bb21416bbbd3619890b74da67430c5e413/appengine/findit/services/test_failure/ci_test_failure.py [modify] https://crrev.com/2fb7d5bb21416bbbd3619890b74da67430c5e413/appengine/findit/libs/test_results/gtest_test_results.py [modify] https://crrev.com/2fb7d5bb21416bbbd3619890b74da67430c5e413/appengine/findit/libs/test_results/test/gtest_test_results_test.py [modify] https://crrev.com/2fb7d5bb21416bbbd3619890b74da67430c5e413/appengine/findit/libs/test_results/base_test_results.py [add] https://crrev.com/2fb7d5bb21416bbbd3619890b74da67430c5e413/appengine/findit/libs/test_results/classified_test_results.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/f0e89bb55b2eb7bb199797132e9795a056686d9c commit f0e89bb55b2eb7bb199797132e9795a056686d9c Author: Chan <chanli@chromium.org> Date: Tue May 29 20:12:27 2018 [Findit] Use the new processed test results to deflake and determine pass rate. In this CL, use the new processed test results from https://chromium-review.googlesource.com/c/infra/infra/+/1067885 to deflake and determine pass rate. Also remove the deprecated GetTestsRunStatuses from test result parser. Bug: 806002 Change-Id: I116fe021d72b5652f84b2f0baded5d7aa1710cfb Reviewed-on: https://chromium-review.googlesource.com/1067937 Commit-Queue: Chan Li <chanli@chromium.org> Reviewed-by: Shuotao Gao <stgao@chromium.org> Reviewed-by: Jeffrey Li <lijeffrey@chromium.org> [modify] https://crrev.com/f0e89bb55b2eb7bb199797132e9795a056686d9c/appengine/findit/libs/test_results/test/classified_test_results_test.py [modify] https://crrev.com/f0e89bb55b2eb7bb199797132e9795a056686d9c/appengine/findit/services/flake_failure/test/flake_swarming_test.py [modify] https://crrev.com/f0e89bb55b2eb7bb199797132e9795a056686d9c/appengine/findit/libs/test_results/gtest_test_results.py [modify] https://crrev.com/f0e89bb55b2eb7bb199797132e9795a056686d9c/appengine/findit/model/test/wf_swarming_task_test.py [modify] https://crrev.com/f0e89bb55b2eb7bb199797132e9795a056686d9c/appengine/findit/libs/test_results/test/gtest_test_results_test.py [add] https://crrev.com/f0e89bb55b2eb7bb199797132e9795a056686d9c/appengine/findit/services/flake_failure/test/flake_test_results_test.py [modify] https://crrev.com/f0e89bb55b2eb7bb199797132e9795a056686d9c/appengine/findit/libs/test_results/base_test_results.py [modify] https://crrev.com/f0e89bb55b2eb7bb199797132e9795a056686d9c/appengine/findit/services/test_failure/test_swarming.py [add] https://crrev.com/f0e89bb55b2eb7bb199797132e9795a056686d9c/appengine/findit/services/flake_failure/flake_test_results.py [modify] https://crrev.com/f0e89bb55b2eb7bb199797132e9795a056686d9c/appengine/findit/model/wf_swarming_task.py [modify] https://crrev.com/f0e89bb55b2eb7bb199797132e9795a056686d9c/appengine/findit/services/test_failure/test/test_swarming_test.py [modify] https://crrev.com/f0e89bb55b2eb7bb199797132e9795a056686d9c/appengine/findit/services/flake_failure/flake_swarming.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/021d8c3a4f8d5a526aa7e2103aa327f362ecab4b commit 021d8c3a4f8d5a526aa7e2103aa327f362ecab4b Author: Chan <chanli@chromium.org> Date: Tue May 29 20:26:27 2018 [Findit] New parser for isolated-script-tests results log. Currently this parser will only parse output.json. Please refer to https://chromium.googlesource.com/chromium/src/+/master/docs/testing/json_test_results_format.md for the format. Bug: 806002 Change-Id: I826441f30cf6f7dd237c5039d0a6b7d98ced11cb Reviewed-on: https://chromium-review.googlesource.com/1000717 Commit-Queue: Chan Li <chanli@chromium.org> Reviewed-by: Shuotao Gao <stgao@chromium.org> Reviewed-by: Roberto Carrillo <robertocn@chromium.org> Reviewed-by: Sean McCullough <seanmccullough@chromium.org> [add] https://crrev.com/021d8c3a4f8d5a526aa7e2103aa327f362ecab4b/appengine/findit/libs/test_results/test/webkit_layout_test_results_test.py [add] https://crrev.com/021d8c3a4f8d5a526aa7e2103aa327f362ecab4b/appengine/findit/libs/test_results/webkit_layout_test_results.py [modify] https://crrev.com/021d8c3a4f8d5a526aa7e2103aa327f362ecab4b/appengine/findit/pipelines/flake_failure/run_flake_swarming_task_pipeline.py [modify] https://crrev.com/021d8c3a4f8d5a526aa7e2103aa327f362ecab4b/appengine/findit/model/test/wf_swarming_task_test.py [modify] https://crrev.com/021d8c3a4f8d5a526aa7e2103aa327f362ecab4b/appengine/findit/services/test_failure/ci_test_failure.py [modify] https://crrev.com/021d8c3a4f8d5a526aa7e2103aa327f362ecab4b/appengine/findit/libs/test_results/gtest_test_results.py [modify] https://crrev.com/021d8c3a4f8d5a526aa7e2103aa327f362ecab4b/appengine/findit/libs/test_results/test_results_util.py [modify] https://crrev.com/021d8c3a4f8d5a526aa7e2103aa327f362ecab4b/appengine/findit/services/flake_failure/test/flake_test_results_test.py [modify] https://crrev.com/021d8c3a4f8d5a526aa7e2103aa327f362ecab4b/appengine/findit/services/flake_failure/test/flake_swarming_test.py [modify] https://crrev.com/021d8c3a4f8d5a526aa7e2103aa327f362ecab4b/appengine/findit/libs/test_results/test/gtest_test_results_test.py [modify] https://crrev.com/021d8c3a4f8d5a526aa7e2103aa327f362ecab4b/appengine/findit/pipelines/flake_failure/test/run_flake_swarming_task_pipeline_test.py [modify] https://crrev.com/021d8c3a4f8d5a526aa7e2103aa327f362ecab4b/appengine/findit/services/test_failure/test/extract_test_signal_test.py [modify] https://crrev.com/021d8c3a4f8d5a526aa7e2103aa327f362ecab4b/appengine/findit/services/flake_failure/flake_swarming.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/e138ec998755e73b1d7d00c53e1459ade7f60f80 commit e138ec998755e73b1d7d00c53e1459ade7f60f80 Author: Chan <chanli@chromium.org> Date: Fri Jun 01 22:52:38 2018 [Findit] Filter out test steps that are isolated-script-tests but not webkit_layout_tests. Currently the new parser makes Findit understand json test results for all isolated-script-tests, but some necessary flags are only added to webkit_layout_tests. To make sure Findit doesn't try to trigger swarming task or try job on non-webkit_layout_tests test steps, filter them out at the very beginning of heuristic analysis. Bug: 806002 Change-Id: Id51adbcf97433f531c11d70c99ddc75aecd0f578 Reviewed-on: https://chromium-review.googlesource.com/1083417 Commit-Queue: Chan Li <chanli@chromium.org> Reviewed-by: Shuotao Gao <stgao@chromium.org> [modify] https://crrev.com/e138ec998755e73b1d7d00c53e1459ade7f60f80/appengine/findit/services/step_util.py [modify] https://crrev.com/e138ec998755e73b1d7d00c53e1459ade7f60f80/appengine/findit/waterfall/flake/step_mapper.py [modify] https://crrev.com/e138ec998755e73b1d7d00c53e1459ade7f60f80/appengine/findit/services/ci_failure.py [modify] https://crrev.com/e138ec998755e73b1d7d00c53e1459ade7f60f80/appengine/findit/services/test_failure/ci_test_failure.py [modify] https://crrev.com/e138ec998755e73b1d7d00c53e1459ade7f60f80/appengine/findit/services/test/step_util_test.py [modify] https://crrev.com/e138ec998755e73b1d7d00c53e1459ade7f60f80/appengine/findit/services/test/ci_failure_test.py [modify] https://crrev.com/e138ec998755e73b1d7d00c53e1459ade7f60f80/appengine/findit/common/constants.py [modify] https://crrev.com/e138ec998755e73b1d7d00c53e1459ade7f60f80/appengine/findit/services/test_failure/test/ci_test_failure_test.py
Ecosystem infra rotation ping! Looks like there was good progress on this back in May. chanli@chromium.org is this still on your radar?
(chanli@ is on vacation) As of this bug, I think it is done because the initial support for webkit layout test seems to work as expected (example linked below. We haven't announced this yet), although minor fix might be needed here and there. So I'd mark it as fixed. https://findit-for-me.appspot.com/waterfall/failure?url=https://luci-milo.appspot.com/buildbot/chromium.webkit/WebKit%20Win10/38697 https://findit-for-me.appspot.com/waterfall/flake?key=ag9zfmZpbmRpdC1mb3ItbWVy6gELEhdNYXN0ZXJGbGFrZUFuYWx5c2lzUm9vdCKzAWNocm9taXVtLm1hYy9NYWMxMC4xMyBUZXN0cyAoZGJnKS80NDQ1L3dlYmtpdF9sYXlvdXRfdGVzdHMvZG1seWRIVmhiQzl3WVdsdWRDMTBiM1ZqYUdGamRHbHZiaTF5WldOMGN5OW1ZWE4wTDJWMlpXNTBjeTkwYjNWamFDOTBiM1ZqYUMxc1lYUmphR1ZrTFhOamNtOXNiQzF1YjJSbExYSmxiVzkyWldRdWFIUnRiQT09DAsSE01hc3RlckZsYWtlQW5hbHlzaXMYAQw
Comment 1 by foolip@chromium.org
, Mar 28 2018