For the perf use case (https://docs.google.com/document/d/1aZh0kKgt3oFO7V3a-LFE3osJAtXVpapO-2o6VI5vQd4/edit) and potentially the webrtc use case (https://docs.google.com/document/d/1v8E-WIExGxelMEzFkbVlHfY99M0Tc_cXPm_BfY1GCX0/edit#heading=h.azfw91mznpm1) we need the ability to grep a directory for files produced from swarming bots. These files include, but are not limited to, output.json and perf-output.json. Therefore, we are asking to change the api to the custom merge script called from swarmings collect_task.py. https://cs.chromium.org/chromium/build/scripts/slave/recipe_modules/swarming/resources/collect_task.py?q=collect_task&sq=package:chromium&l=35 We would like this to be a path to a directory that we can traverse instead of a set of file paths.
on review: https://chromium-review.googlesource.com/c/chromium/tools/build/+/845781 to pass --task-output-dir to merge scripts https://chromium-review.googlesource.com/c/chromium/src/+/845780 to have existing merge scripts accept --task-output-dir
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/826d5edaf7c3d1a935485350bbfa006839df6da1 commit 826d5edaf7c3d1a935485350bbfa006839df6da1 Author: John Budorick <jbudorick@chromium.org> Date: Thu Dec 28 21:16:24 2017 Use merge & --gs-results-bucket consistently on Android bots. I'm making some changes to the merge script API for crbug.com/793359 and figured I'd clean this up while I'm in the neighborhood. This CL: - removes all existing merge exceptions - Uses --gs-results-bucket for all swarmed Android tests. - BBGenErr can now take an exception as a cause. If present, the cause's stack trace will be added to the BBGenErr's message. I've used this for handling errors in generation. - check_output_file_consistency will now indicate what differs between actual and expected Bug: 793359 Change-Id: I2892e2431a1d325e2a4137ba799ce4ac5b547ced Reviewed-on: https://chromium-review.googlesource.com/845779 Commit-Queue: John Budorick <jbudorick@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#526326} [modify] https://crrev.com/826d5edaf7c3d1a935485350bbfa006839df6da1/testing/buildbot/chromium.android.fyi.json [modify] https://crrev.com/826d5edaf7c3d1a935485350bbfa006839df6da1/testing/buildbot/chromium.android.json [modify] https://crrev.com/826d5edaf7c3d1a935485350bbfa006839df6da1/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/826d5edaf7c3d1a935485350bbfa006839df6da1/testing/buildbot/generate_buildbot_json.py [modify] https://crrev.com/826d5edaf7c3d1a935485350bbfa006839df6da1/testing/buildbot/generate_buildbot_json_unittest.py [modify] https://crrev.com/826d5edaf7c3d1a935485350bbfa006839df6da1/testing/buildbot/test_suite_exceptions.pyl [modify] https://crrev.com/826d5edaf7c3d1a935485350bbfa006839df6da1/testing/buildbot/test_suites.pyl
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/e6b01c7585ad85e5919a33d52f4d16e352e0b97e commit e6b01c7585ad85e5919a33d52f4d16e352e0b97e Author: John Budorick <jbudorick@chromium.org> Date: Wed Jan 03 21:18:59 2018 Add --task-output-dir to existing merge scripts. Bug: 793359 Change-Id: I874c3ed12d9de938909d89a394ca6929f5d1f494 Reviewed-on: https://chromium-review.googlesource.com/845780 Reviewed-by: Quinten Yearsley <qyearsley@chromium.org> Reviewed-by: Emily Hanley <eyaich@chromium.org> Commit-Queue: John Budorick <jbudorick@chromium.org> Cr-Commit-Position: refs/heads/master@{#526817} [modify] https://crrev.com/e6b01c7585ad85e5919a33d52f4d16e352e0b97e/build/android/pylib/results/presentation/test_results_presentation.py [modify] https://crrev.com/e6b01c7585ad85e5919a33d52f4d16e352e0b97e/third_party/WebKit/Tools/Scripts/merge-layout-test-results [modify] https://crrev.com/e6b01c7585ad85e5919a33d52f4d16e352e0b97e/tools/perf/upload_perf_results.py
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/dcbcf261381f944fa12fd470427633d32f7f785d commit dcbcf261381f944fa12fd470427633d32f7f785d Author: John Budorick <jbudorick@chromium.org> Date: Wed Jan 03 22:04:55 2018 Pass --task-output-dir to merge scripts. Bug: 793359 Change-Id: I5a937d6a338f4ff30959f13bf0ff11688e5b13e9 Reviewed-on: https://chromium-review.googlesource.com/845781 Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> Reviewed-by: Emily Hanley <eyaich@chromium.org> Commit-Queue: John Budorick <jbudorick@chromium.org> [modify] https://crrev.com/dcbcf261381f944fa12fd470427633d32f7f785d/scripts/slave/recipe_modules/swarming/unittests/collect_task_test.py [modify] https://crrev.com/dcbcf261381f944fa12fd470427633d32f7f785d/scripts/slave/recipe_modules/swarming/resources/collect_task.py
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/49df9d4aa3e0db3b9655a471eac0f7ed7d51756f commit 49df9d4aa3e0db3b9655a471eac0f7ed7d51756f Author: John Budorick <jbudorick@chromium.org> Date: Wed Jan 03 22:55:36 2018 Revert "Pass --task-output-dir to merge scripts." This reverts commit dcbcf261381f944fa12fd470427633d32f7f785d. Reason for revert: X( forgot to update the standard merge scripts. Original change's description: > Pass --task-output-dir to merge scripts. > > Bug: 793359 > Change-Id: I5a937d6a338f4ff30959f13bf0ff11688e5b13e9 > Reviewed-on: https://chromium-review.googlesource.com/845781 > Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> > Reviewed-by: Emily Hanley <eyaich@chromium.org> > Commit-Queue: John Budorick <jbudorick@chromium.org> TBR=maruel@chromium.org,eyaich@chromium.org,jbudorick@chromium.org Change-Id: Ie918b4150d7da45a0875081f0d10a346b23ebf4e No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 793359 Reviewed-on: https://chromium-review.googlesource.com/849413 Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: John Budorick <jbudorick@chromium.org> [modify] https://crrev.com/49df9d4aa3e0db3b9655a471eac0f7ed7d51756f/scripts/slave/recipe_modules/swarming/unittests/collect_task_test.py [modify] https://crrev.com/49df9d4aa3e0db3b9655a471eac0f7ed7d51756f/scripts/slave/recipe_modules/swarming/resources/collect_task.py
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/ea3d60db09215ea3ec8d74f6aa13e2d9da075e58 commit ea3d60db09215ea3ec8d74f6aa13e2d9da075e58 Author: John Budorick <jbudorick@chromium.org> Date: Mon Jan 08 15:45:03 2018 Pass --task-output-dir to merge scripts. (RELAND) Bug: 793359 Change-Id: I7b1a1b896e6989e1c4c6c904ee88e8054462acb9 Reviewed-on: https://chromium-review.googlesource.com/849592 Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> Commit-Queue: John Budorick <jbudorick@chromium.org> [modify] https://crrev.com/ea3d60db09215ea3ec8d74f6aa13e2d9da075e58/scripts/slave/recipe_modules/swarming/resources/standard_isolated_script_merge.py [modify] https://crrev.com/ea3d60db09215ea3ec8d74f6aa13e2d9da075e58/scripts/slave/recipe_modules/swarming/resources/standard_gtest_merge.py [modify] https://crrev.com/ea3d60db09215ea3ec8d74f6aa13e2d9da075e58/scripts/slave/recipe_modules/swarming/unittests/collect_task_test.py [modify] https://crrev.com/ea3d60db09215ea3ec8d74f6aa13e2d9da075e58/scripts/slave/recipe_modules/swarming/resources/collect_task.py
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/c53130aa9b4e1ebffbf2765f150d2e480bd2654d commit c53130aa9b4e1ebffbf2765f150d2e480bd2654d Author: John Budorick <jbudorick@chromium.org> Date: Mon Jan 08 16:44:04 2018 Revert "Pass --task-output-dir to merge scripts. (RELAND)" This reverts commit ea3d60db09215ea3ec8d74f6aa13e2d9da075e58. Reason for revert: broke iOS. sigh. Original change's description: > Pass --task-output-dir to merge scripts. (RELAND) > > Bug: 793359 > Change-Id: I7b1a1b896e6989e1c4c6c904ee88e8054462acb9 > Reviewed-on: https://chromium-review.googlesource.com/849592 > Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> > Commit-Queue: John Budorick <jbudorick@chromium.org> TBR=maruel@chromium.org,eyaich@chromium.org,jbudorick@chromium.org Change-Id: I5ed70794ee1d966a1fddd8d20473d0f7d4737a0e No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 793359 Reviewed-on: https://chromium-review.googlesource.com/854235 Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: John Budorick <jbudorick@chromium.org> [modify] https://crrev.com/c53130aa9b4e1ebffbf2765f150d2e480bd2654d/scripts/slave/recipe_modules/swarming/resources/standard_isolated_script_merge.py [modify] https://crrev.com/c53130aa9b4e1ebffbf2765f150d2e480bd2654d/scripts/slave/recipe_modules/swarming/resources/standard_gtest_merge.py [modify] https://crrev.com/c53130aa9b4e1ebffbf2765f150d2e480bd2654d/scripts/slave/recipe_modules/swarming/unittests/collect_task_test.py [modify] https://crrev.com/c53130aa9b4e1ebffbf2765f150d2e480bd2654d/scripts/slave/recipe_modules/swarming/resources/collect_task.py
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/c5f04477a88e82392b1a9f1c4515bcacc96490b5 commit c5f04477a88e82392b1a9f1c4515bcacc96490b5 Author: John Budorick <jbudorick@chromium.org> Date: Mon Jan 08 20:07:42 2018 Pass --task-output-dir to merge scripts. (RELAND 2) This extracts ArgumentParser creation for the three merge scripts in the swarming recipe_module to a common module. It also adds a test for each of the three merge scripts to verify that they accept --task-output-dir. Bug: 793359 Change-Id: I528920a9e501aca8e8bffcc2a79bdb2bf0a86612 Reviewed-on: https://chromium-review.googlesource.com/854472 Commit-Queue: John Budorick <jbudorick@chromium.org> Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org> [modify] https://crrev.com/c5f04477a88e82392b1a9f1c4515bcacc96490b5/scripts/slave/recipe_modules/swarming/unittests/noop_merge_test.py [modify] https://crrev.com/c5f04477a88e82392b1a9f1c4515bcacc96490b5/scripts/slave/recipe_modules/swarming/resources/standard_gtest_merge.py [modify] https://crrev.com/c5f04477a88e82392b1a9f1c4515bcacc96490b5/scripts/slave/recipe_modules/swarming/unittests/standard_gtest_merge_test.py [modify] https://crrev.com/c5f04477a88e82392b1a9f1c4515bcacc96490b5/scripts/slave/recipe_modules/swarming/unittests/standard_isolated_script_merge_test.py [modify] https://crrev.com/c5f04477a88e82392b1a9f1c4515bcacc96490b5/scripts/slave/recipe_modules/swarming/resources/noop_merge.py [modify] https://crrev.com/c5f04477a88e82392b1a9f1c4515bcacc96490b5/scripts/slave/recipe_modules/swarming/unittests/collect_task_test.py [add] https://crrev.com/c5f04477a88e82392b1a9f1c4515bcacc96490b5/scripts/slave/recipe_modules/swarming/unittests/common_merge_script_tests.py [modify] https://crrev.com/c5f04477a88e82392b1a9f1c4515bcacc96490b5/scripts/slave/recipe_modules/swarming/resources/standard_isolated_script_merge.py [add] https://crrev.com/c5f04477a88e82392b1a9f1c4515bcacc96490b5/scripts/slave/recipe_modules/swarming/resources/merge_api.py [modify] https://crrev.com/c5f04477a88e82392b1a9f1c4515bcacc96490b5/scripts/slave/recipe_modules/swarming/resources/collect_task.py
Comment 1 by bpastene@chromium.org
, Dec 8 2017Status: Assigned (was: Untriaged)