1. Move logics to service modules 2. Move pipelines to pipeline modules 3. Make pipelines to inherit from new basepipelines
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/1f447fb828c96fa2ab39fc839ce6d61f0e3f4712 commit 1f447fb828c96fa2ab39fc839ce6d61f0e3f4712 Author: Chan <chanli@chromium.org> Date: Tue Mar 20 16:41:46 2018 [Findit] Add new endpoint to receive swarming pubsub notifications. This is preparation for using new pubsub logic for running swarming task pipelines. Bug: 808695 Change-Id: Ide7272ccf941171d6a8216ae54fbcf0ec5db7df7 Reviewed-on: https://chromium-review.googlesource.com/969759 Reviewed-by: Shuotao Gao <stgao@chromium.org> Commit-Queue: Chan Li <chanli@chromium.org> [add] https://crrev.com/1f447fb828c96fa2ab39fc839ce6d61f0e3f4712/appengine/findit/handlers/test/swarming_pubsub_pipeline_callback_test.py [modify] https://crrev.com/1f447fb828c96fa2ab39fc839ce6d61f0e3f4712/appengine/findit/main.py [modify] https://crrev.com/1f447fb828c96fa2ab39fc839ce6d61f0e3f4712/appengine/findit/handlers/test/try_job_pubsub_pipeline_callback_test.py [add] https://crrev.com/1f447fb828c96fa2ab39fc839ce6d61f0e3f4712/appengine/findit/handlers/swarming_pubsub_pipeline_callback.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/d2a8039dc2ce890a979eaa7e00ed877d592fd82e commit d2a8039dc2ce890a979eaa7e00ed877d592fd82e Author: Chan <chanli@chromium.org> Date: Tue Mar 27 00:52:12 2018 [Findit] dto for run test swarming task pipeline input. Bug: 808695 Change-Id: Ic8f51ecfe5f9a4e2d0ca9666b1d7f5ee2424af66 Reviewed-on: https://chromium-review.googlesource.com/978542 Commit-Queue: Chan Li <chanli@chromium.org> Reviewed-by: Roberto Carrillo <robertocn@chromium.org> [add] https://crrev.com/d2a8039dc2ce890a979eaa7e00ed877d592fd82e/appengine/findit/dto/run_swarming_task_parameters.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/2d2af45ad59d0343ecd572693f28abfcd231c91e commit 2d2af45ad59d0343ecd572693f28abfcd231c91e Author: Chan <chanli@chromium.org> Date: Tue Mar 27 19:19:13 2018 [Findit] Separate logic of triggering swarming task to service module. This is the first CL to refactor pipelines of triggering swarming task. What's changed: - In this CL, general logic of triggering swarming tasks and - triggering reruns for waterfall test failures are moved to service module. What's not changed: - Don't change the logic to adapt to the new pubsub-based callback, will do it in a following CL. - Don't hook pipelines with the separated logic and remove deprecated pipelines, will do it in a following CL. - Because the ongoing refactor at FA side, special logic of triggering swarming tasks for FA was not touched. Bug: 808695 Change-Id: I351bb448c829a249104815daa1b31180b81b2ec4 Reviewed-on: https://chromium-review.googlesource.com/966930 Commit-Queue: Chan Li <chanli@chromium.org> Reviewed-by: Jeffrey Li <lijeffrey@chromium.org> Reviewed-by: Roberto Carrillo <robertocn@chromium.org> [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/pipelines/flake_failure/run_flake_try_job_pipeline.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/waterfall/test/trigger_base_swarming_task_pipeline_test.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/infra_api_clients/swarming/swarming_util.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/dto/step_metadata.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/services/test/swarming_test.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/services/test/monitoring_test.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/services/monitoring.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/pipelines/flake_failure/test/run_flake_try_job_pipeline_test.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/waterfall/test/trigger_flake_swarming_task_pipeline_test.py [delete] https://crrev.com/22ec2b82eeef041ab22ecd315b7fe0b96363e57b/appengine/findit/dto/swarming_task_request.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/waterfall/test/wf_testcase.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/pipelines/flake_failure/get_isolate_sha_pipeline.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/infra_api_clients/swarming/test/swarming_util_test.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/services/swarming.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/infra_api_clients/swarming/swarming_task_request.py [add] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/services/test_failure/test_swarming.py [delete] https://crrev.com/22ec2b82eeef041ab22ecd315b7fe0b96363e57b/appengine/findit/dto/test/swarming_task_request_test.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/services/swarmed_test_util.py [add] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/services/test_failure/test/test_swarming_test.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/services/step_util.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/waterfall/trigger_base_swarming_task_pipeline.py [rename] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/libs/list_of_basestring.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/services/test/swarmed_test_util_test.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/infra_api_clients/swarming/test/swarming_task_request_test.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/services/flake_failure/test/flake_try_job_test.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/waterfall/trigger_flake_swarming_task_pipeline.py [modify] https://crrev.com/2d2af45ad59d0343ecd572693f28abfcd231c91e/appengine/findit/pipelines/flake_failure/test/get_isolate_sha_pipeline_test.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/4ba8dd932dd8e78cbdb5c2b00694c5e030d15240 commit 4ba8dd932dd8e78cbdb5c2b00694c5e030d15240 Author: Chan <chanli@chromium.org> Date: Tue Mar 27 19:52:24 2018 [Findit] Refactor: New pipeline to run swarming rerun to deflake. This CL adds a new pipeline to trigger and process a swarming rerun for a step. Bug: 808695 Change-Id: I4086a1cbc90da3859ea4d2115344744a6356fe73 Reviewed-on: https://chromium-review.googlesource.com/978544 Reviewed-by: Jeffrey Li <lijeffrey@chromium.org> Commit-Queue: Chan Li <chanli@chromium.org> [add] https://crrev.com/4ba8dd932dd8e78cbdb5c2b00694c5e030d15240/appengine/findit/pipelines/test_failure/run_test_swarming_task_pipeline.py [modify] https://crrev.com/4ba8dd932dd8e78cbdb5c2b00694c5e030d15240/appengine/findit/services/test/swarmed_test_util_test.py [modify] https://crrev.com/4ba8dd932dd8e78cbdb5c2b00694c5e030d15240/appengine/findit/waterfall/trigger_base_swarming_task_pipeline.py [modify] https://crrev.com/4ba8dd932dd8e78cbdb5c2b00694c5e030d15240/appengine/findit/waterfall/process_base_swarming_task_result_pipeline.py [modify] https://crrev.com/4ba8dd932dd8e78cbdb5c2b00694c5e030d15240/appengine/findit/waterfall/trigger_flake_swarming_task_pipeline.py [add] https://crrev.com/4ba8dd932dd8e78cbdb5c2b00694c5e030d15240/appengine/findit/pipelines/test_failure/test/run_test_swarming_task_pipeline_test.py [modify] https://crrev.com/4ba8dd932dd8e78cbdb5c2b00694c5e030d15240/appengine/findit/services/constants.py [modify] https://crrev.com/4ba8dd932dd8e78cbdb5c2b00694c5e030d15240/appengine/findit/dto/swarming_task_error.py [modify] https://crrev.com/4ba8dd932dd8e78cbdb5c2b00694c5e030d15240/appengine/findit/services/test_failure/test_swarming.py [modify] https://crrev.com/4ba8dd932dd8e78cbdb5c2b00694c5e030d15240/appengine/findit/waterfall/trigger_swarming_task_pipeline.py [modify] https://crrev.com/4ba8dd932dd8e78cbdb5c2b00694c5e030d15240/appengine/findit/services/swarmed_test_util.py [modify] https://crrev.com/4ba8dd932dd8e78cbdb5c2b00694c5e030d15240/appengine/findit/services/test_failure/test/test_swarming_test.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/bade247437b6fc3ee8a8c5c4f1559d09d8ed7ea9 commit bade247437b6fc3ee8a8c5c4f1559d09d8ed7ea9 Author: Chan <chanli@chromium.org> Date: Thu Mar 29 20:33:22 2018 [Findit] Wrapper pipeline to run swarming tasks. Wrapper pipeline that spawns swarming task pipelines for new failed steps in a build. This pipeline will get all first time failed test steps that have not been deflaked then run swarming reruns for each step. Bug: 808695 Change-Id: I879b8685cad6321b9013fb7b7afba10df967aa4a Reviewed-on: https://chromium-review.googlesource.com/979596 Commit-Queue: Chan Li <chanli@chromium.org> Reviewed-by: Shuotao Gao <stgao@chromium.org> Reviewed-by: Roberto Carrillo <robertocn@chromium.org> [add] https://crrev.com/bade247437b6fc3ee8a8c5c4f1559d09d8ed7ea9/appengine/findit/pipelines/test_failure/run_swarming_tasks_pipeline.py [modify] https://crrev.com/bade247437b6fc3ee8a8c5c4f1559d09d8ed7ea9/appengine/findit/pipelines/test_failure/test/run_test_swarming_task_pipeline_test.py [modify] https://crrev.com/bade247437b6fc3ee8a8c5c4f1559d09d8ed7ea9/appengine/findit/pipelines/test_failure/run_test_swarming_task_pipeline.py [modify] https://crrev.com/bade247437b6fc3ee8a8c5c4f1559d09d8ed7ea9/appengine/findit/services/test/swarmed_test_util_test.py [modify] https://crrev.com/bade247437b6fc3ee8a8c5c4f1559d09d8ed7ea9/appengine/findit/services/test_failure/test_failure_analysis.py [modify] https://crrev.com/bade247437b6fc3ee8a8c5c4f1559d09d8ed7ea9/appengine/findit/waterfall/flake/step_mapper.py [add] https://crrev.com/bade247437b6fc3ee8a8c5c4f1559d09d8ed7ea9/appengine/findit/dto/run_swarming_tasks_input.py [modify] https://crrev.com/bade247437b6fc3ee8a8c5c4f1559d09d8ed7ea9/appengine/findit/dto/swarming_task_error.py [modify] https://crrev.com/bade247437b6fc3ee8a8c5c4f1559d09d8ed7ea9/appengine/findit/services/test_failure/test/test_failure_analysis_test.py [modify] https://crrev.com/bade247437b6fc3ee8a8c5c4f1559d09d8ed7ea9/appengine/findit/services/swarmed_test_util.py [modify] https://crrev.com/bade247437b6fc3ee8a8c5c4f1559d09d8ed7ea9/appengine/findit/services/test_failure/test/test_swarming_test.py [add] https://crrev.com/bade247437b6fc3ee8a8c5c4f1559d09d8ed7ea9/appengine/findit/pipelines/test_failure/test/run_swarming_tasks_pipeline_test.py [modify] https://crrev.com/bade247437b6fc3ee8a8c5c4f1559d09d8ed7ea9/appengine/findit/services/test_failure/test_swarming.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/4453fb7bd1289667d38e7ff37f1ae408762f5249 commit 4453fb7bd1289667d38e7ff37f1ae408762f5249 Author: Chan <chanli@chromium.org> Date: Thu Mar 29 22:25:32 2018 [Findit] Update WfSwarmingTask for timestamps when swarming task completes. Bug: 808695 Change-Id: I8f5382bd49b87f5d87243a15c9378a7b4617b252 Reviewed-on: https://chromium-review.googlesource.com/986722 Reviewed-by: Jeffrey Li <lijeffrey@chromium.org> Commit-Queue: Chan Li <chanli@chromium.org> [modify] https://crrev.com/4453fb7bd1289667d38e7ff37f1ae408762f5249/appengine/findit/services/test/swarmed_test_util_test.py [modify] https://crrev.com/4453fb7bd1289667d38e7ff37f1ae408762f5249/appengine/findit/services/swarmed_test_util.py [modify] https://crrev.com/4453fb7bd1289667d38e7ff37f1ae408762f5249/appengine/findit/services/test_failure/test/test_swarming_test.py [modify] https://crrev.com/4453fb7bd1289667d38e7ff37f1ae408762f5249/appengine/findit/services/test_failure/test_swarming.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/d7b96b1105d2bbece773b8858fc96f515410e58a commit d7b96b1105d2bbece773b8858fc96f515410e58a Author: Chan <chanli@chromium.org> Date: Fri Mar 30 01:13:05 2018 [Findit] Add an async pipeline to periodically check if all swarming tasks have completed. This is an async pipeline which will do self callback periodically to check if all swarming tasks have completed. This pipeline will provide consistent failures to test_try_job_pipeline to run try job. Bug: 808695 Change-Id: I983d0b0b5bde912bd71f6fbe2082d0174e099959 Reviewed-on: https://chromium-review.googlesource.com/981606 Commit-Queue: Chan Li <chanli@chromium.org> Reviewed-by: Shuotao Gao <stgao@chromium.org> [modify] https://crrev.com/d7b96b1105d2bbece773b8858fc96f515410e58a/appengine/findit/services/test_failure/test_failure_analysis.py [add] https://crrev.com/d7b96b1105d2bbece773b8858fc96f515410e58a/appengine/findit/dto/collect_swarming_task_results_inputs.py [add] https://crrev.com/d7b96b1105d2bbece773b8858fc96f515410e58a/appengine/findit/pipelines/test_failure/test/collect_swarming_task_results_pipeline_test.py [add] https://crrev.com/d7b96b1105d2bbece773b8858fc96f515410e58a/appengine/findit/pipelines/test_failure/collect_swarming_task_results_pipeline.py [modify] https://crrev.com/d7b96b1105d2bbece773b8858fc96f515410e58a/appengine/findit/services/test_failure/test/test_swarming_test.py [modify] https://crrev.com/d7b96b1105d2bbece773b8858fc96f515410e58a/appengine/findit/services/test_failure/test_swarming.py [add] https://crrev.com/d7b96b1105d2bbece773b8858fc96f515410e58a/appengine/findit/dto/collect_swarming_task_results_outputs.py [modify] https://crrev.com/d7b96b1105d2bbece773b8858fc96f515410e58a/appengine/findit/services/test_failure/ci_test_failure.py [modify] https://crrev.com/d7b96b1105d2bbece773b8858fc96f515410e58a/appengine/findit/services/test_failure/test/test_failure_analysis_test.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/7e035ef01dfc3a99ed1a2e2bb08d0c7d196710a4 commit 7e035ef01dfc3a99ed1a2e2bb08d0c7d196710a4 Author: Chan <chanli@chromium.org> Date: Tue Apr 03 02:47:47 2018 [Findit] Use new or refactoring pipelines to run swarming tasks to deflake. This CL replaces the old TriggerSwarmingTasksPipeline and ProcessSwarmingTasksResultPipeline with the new pipelines. Bug: 808695 Change-Id: Ie5621f3f62cf89f87d438aad99a800f450fb4ecd Reviewed-on: https://chromium-review.googlesource.com/982939 Commit-Queue: Chan Li <chanli@chromium.org> Reviewed-by: Shuotao Gao <stgao@chromium.org> [modify] https://crrev.com/7e035ef01dfc3a99ed1a2e2bb08d0c7d196710a4/appengine/findit/services/compile_failure/test/compile_try_job_test.py [modify] https://crrev.com/7e035ef01dfc3a99ed1a2e2bb08d0c7d196710a4/appengine/findit/pipelines/test_failure/analyze_test_failure_pipeline.py [modify] https://crrev.com/7e035ef01dfc3a99ed1a2e2bb08d0c7d196710a4/appengine/findit/services/test/try_job_test.py [modify] https://crrev.com/7e035ef01dfc3a99ed1a2e2bb08d0c7d196710a4/appengine/findit/pipelines/test_failure/start_test_try_job_pipeline.py [modify] https://crrev.com/7e035ef01dfc3a99ed1a2e2bb08d0c7d196710a4/appengine/findit/pipelines/test_failure/test/start_test_try_job_pipeline_test.py [add] https://crrev.com/7e035ef01dfc3a99ed1a2e2bb08d0c7d196710a4/appengine/findit/dto/start_try_job_inputs.py [modify] https://crrev.com/7e035ef01dfc3a99ed1a2e2bb08d0c7d196710a4/appengine/findit/pipelines/test_failure/test/analyze_test_failure_pipeline_test.py [modify] https://crrev.com/7e035ef01dfc3a99ed1a2e2bb08d0c7d196710a4/appengine/findit/services/test_failure/test/test_try_job_test.py [modify] https://crrev.com/7e035ef01dfc3a99ed1a2e2bb08d0c7d196710a4/appengine/findit/services/try_job.py [modify] https://crrev.com/7e035ef01dfc3a99ed1a2e2bb08d0c7d196710a4/appengine/findit/services/test_failure/test_try_job.py
The following revision refers to this bug: https://chromium.googlesource.com/infra/infra/+/e846a59e7e587f429e88076701545618a7df52e5 commit e846a59e7e587f429e88076701545618a7df52e5 Author: Chan <chanli@chromium.org> Date: Thu Apr 05 23:07:09 2018 [Findit] Update WfAnalysis about flaky tests found by swarming reruns. This is to fix a bug which a feature was missed during refactoring. Copy update_analysis_with_flake_info_pipeline._UpdateAnalysisWithFlakeInfo to test_failure_analysis. update_analysis_with_flake_info_pipeline will be deleted when new pipelines are up and running. Bug: 808695 Change-Id: I4bf8edb02803c9cec13b44696b0c3e48b135e761 Reviewed-on: https://chromium-review.googlesource.com/999035 Reviewed-by: Roberto Carrillo <robertocn@chromium.org> Reviewed-by: Shuotao Gao <stgao@chromium.org> Commit-Queue: Chan Li <chanli@chromium.org> [modify] https://crrev.com/e846a59e7e587f429e88076701545618a7df52e5/appengine/findit/services/test_failure/test_failure_analysis.py [modify] https://crrev.com/e846a59e7e587f429e88076701545618a7df52e5/appengine/findit/services/test_failure/test/test_swarming_test.py [modify] https://crrev.com/e846a59e7e587f429e88076701545618a7df52e5/appengine/findit/services/test_failure/test/test_failure_analysis_test.py [modify] https://crrev.com/e846a59e7e587f429e88076701545618a7df52e5/appengine/findit/services/test_failure/test_swarming.py
Comment 1 by chanli@chromium.org
, Feb 2 2018