It'd be nice to be able to configure a suite to run on a given bot as experiments. Failures should not cause the bot to fail, and the bot should be configurable to run <= 100% of the time.
on review: https://chromium-review.googlesource.com/c/chromium/tools/build/+/860741
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/53d61f9bb3ebf9d098a6aec5a92934780d453a55 commit 53d61f9bb3ebf9d098a6aec5a92934780d453a55 Author: John Budorick <jbudorick@chromium.org> Date: Tue Jan 23 02:54:52 2018 chromium_tests: Extract common test generation logic from generate_gtest. Bug: 797449 Change-Id: Ibc4767c7076341b13fce4b35d299d11c10b9006b Reviewed-on: https://chromium-review.googlesource.com/857691 Commit-Queue: John Budorick <jbudorick@chromium.org> Reviewed-by: Stephen Martinis <martiniss@chromium.org> [modify] https://crrev.com/53d61f9bb3ebf9d098a6aec5a92934780d453a55/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_gtest.expected/invalid tear down.json [modify] https://crrev.com/53d61f9bb3ebf9d098a6aec5a92934780d453a55/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_gtest.expected/trigger_script_invalid.json [modify] https://crrev.com/53d61f9bb3ebf9d098a6aec5a92934780d453a55/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_gtest.py [modify] https://crrev.com/53d61f9bb3ebf9d098a6aec5a92934780d453a55/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_gtest.expected/merge.json [modify] https://crrev.com/53d61f9bb3ebf9d098a6aec5a92934780d453a55/scripts/slave/recipe_modules/chromium_tests/steps.py [modify] https://crrev.com/53d61f9bb3ebf9d098a6aec5a92934780d453a55/scripts/slave/recipes/chromium.expected/gtest_bad_custom_merge_script.json [modify] https://crrev.com/53d61f9bb3ebf9d098a6aec5a92934780d453a55/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_gtest.expected/merge_invalid.json [modify] https://crrev.com/53d61f9bb3ebf9d098a6aec5a92934780d453a55/scripts/slave/recipe_modules/chromium/api.py [modify] https://crrev.com/53d61f9bb3ebf9d098a6aec5a92934780d453a55/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_gtest.expected/invalid set_up.json [modify] https://crrev.com/53d61f9bb3ebf9d098a6aec5a92934780d453a55/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_gtest.expected/trigger_script.json
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/5ba73c236bf285e9da0b1564adb361580e8ae069 commit 5ba73c236bf285e9da0b1564adb361580e8ae069 Author: John Budorick <jbudorick@chromium.org> Date: Fri Feb 02 14:03:11 2018 chromium_tests: Extract common test generation logic from generate_isolated_script. Bug: 797449 Change-Id: Id3ca803924767aa4ba4a886e0cdc5b769a876a4e Reviewed-on: https://chromium-review.googlesource.com/857479 Commit-Queue: John Budorick <jbudorick@chromium.org> Reviewed-by: Stephen Martinis <martiniss@chromium.org> [modify] https://crrev.com/5ba73c236bf285e9da0b1564adb361580e8ae069/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_isolated_script.expected/swarming_trigger_script_invalid.json [modify] https://crrev.com/5ba73c236bf285e9da0b1564adb361580e8ae069/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_isolated_script.expected/bad tear down.json [modify] https://crrev.com/5ba73c236bf285e9da0b1564adb361580e8ae069/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_isolated_script.expected/bad set up.json [modify] https://crrev.com/5ba73c236bf285e9da0b1564adb361580e8ae069/scripts/slave/recipe_modules/chromium_tests/steps.py [modify] https://crrev.com/5ba73c236bf285e9da0b1564adb361580e8ae069/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_isolated_script.expected/merge_invalid.json [modify] https://crrev.com/5ba73c236bf285e9da0b1564adb361580e8ae069/scripts/slave/recipes/chromium.expected/isolated_script_test_bad_custom_merge_script.json
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/f384cd150738f5f23bafd5315ba6470f83fa9838 commit f384cd150738f5f23bafd5315ba6470f83fa9838 Author: John Budorick <jbudorick@chromium.org> Date: Wed Feb 14 23:44:12 2018 chromium_tests: implement experimental tests. Bug: 797449 Change-Id: I68a445aa17d9a5721b1dcf1fb4242368a503f1da Reviewed-on: https://chromium-review.googlesource.com/860741 Commit-Queue: John Budorick <jbudorick@chromium.org> Reviewed-by: Stephen Martinis <martiniss@chromium.org> [modify] https://crrev.com/f384cd150738f5f23bafd5315ba6470f83fa9838/scripts/slave/README.recipes.md [add] https://crrev.com/f384cd150738f5f23bafd5315ba6470f83fa9838/scripts/slave/recipe_modules/chromium_tests/tests/steps/experimental_test.py [modify] https://crrev.com/f384cd150738f5f23bafd5315ba6470f83fa9838/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_isolated_script.py [modify] https://crrev.com/f384cd150738f5f23bafd5315ba6470f83fa9838/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_gtest.py [modify] https://crrev.com/f384cd150738f5f23bafd5315ba6470f83fa9838/scripts/slave/recipe_modules/chromium_tests/steps.py
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/750ba9b2ee7f1678d976b48639a023c58f7f454c commit 750ba9b2ee7f1678d976b48639a023c58f7f454c Author: John Budorick <jbudorick@chromium.org> Date: Fri Feb 16 15:58:18 2018 chromium_tests: Add the remainder of the Test interface to TestWrapper. Also adds a bit of documentation for TestWrapper and ExperimentalTest, and fixes a bug in TestWrapper.name. Bug: 797449 Change-Id: Idd8b829b5c7d8347ecc400337e533d48613dac20 Reviewed-on: https://chromium-review.googlesource.com/920841 Commit-Queue: John Budorick <jbudorick@chromium.org> Reviewed-by: Stephen Martinis <martiniss@chromium.org> [modify] https://crrev.com/750ba9b2ee7f1678d976b48639a023c58f7f454c/scripts/slave/recipe_modules/chromium_tests/tests/steps/experimental_test.py [modify] https://crrev.com/750ba9b2ee7f1678d976b48639a023c58f7f454c/scripts/slave/recipe_modules/chromium_tests/steps.py
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/83cc799fd044a48809db3c7dc6033344c883a183 commit 83cc799fd044a48809db3c7dc6033344c883a183 Author: John Budorick <jbudorick@chromium.org> Date: Fri Feb 16 20:28:09 2018 Ignore has_valid_results and failures for ExperimentalTests. api.test_utils.determine_new_failures uses has_valid_results and failures to determine whether to do the w/o patch retry and whether to add an (actual) failing step to the build. We don't really want either with experimental tests. Bug: 797449 Change-Id: I0f7db6ac6ae02051b4da00fa404b1c28a459072b Reviewed-on: https://chromium-review.googlesource.com/924339 Reviewed-by: Stephen Martinis <martiniss@chromium.org> Commit-Queue: John Budorick <jbudorick@chromium.org> [modify] https://crrev.com/83cc799fd044a48809db3c7dc6033344c883a183/scripts/slave/recipe_modules/chromium_tests/tests/steps/experimental_test.py [modify] https://crrev.com/83cc799fd044a48809db3c7dc6033344c883a183/scripts/slave/recipe_modules/chromium_tests/steps.py
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/354ed5acbaa3f709c5b90b26786e685671307c5e commit 354ed5acbaa3f709c5b90b26786e685671307c5e Author: John Budorick <jbudorick@chromium.org> Date: Tue Feb 20 19:13:39 2018 Add "experimental" to the suffix of experimental tests. Bug: 797449 Change-Id: I4142b9471d689feebbd57491750af72e30b8a38d Reviewed-on: https://chromium-review.googlesource.com/926909 Reviewed-by: Stephen Martinis <martiniss@chromium.org> Commit-Queue: John Budorick <jbudorick@chromium.org> [modify] https://crrev.com/354ed5acbaa3f709c5b90b26786e685671307c5e/scripts/slave/recipe_modules/chromium_tests/tests/steps/generate_gtest.py [modify] https://crrev.com/354ed5acbaa3f709c5b90b26786e685671307c5e/scripts/slave/recipe_modules/chromium_tests/tests/steps/experimental_test.py [modify] https://crrev.com/354ed5acbaa3f709c5b90b26786e685671307c5e/scripts/slave/recipe_modules/chromium_tests/steps.py
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/cdd30e47bed53573a91d9fb42563f3a5d4677484 commit cdd30e47bed53573a91d9fb42563f3a5d4677484 Author: John Budorick <jbudorick@chromium.org> Date: Wed Feb 21 00:21:59 2018 Add documentation for experiment_percentage for tests. Bug: 797449 Change-Id: Ic4d7d360ee322760653d85e9b3e1ba886a3b3441 Reviewed-on: https://chromium-review.googlesource.com/920777 Reviewed-by: Stephen Martinis <martiniss@chromium.org> Reviewed-by: Dirk Pranke <dpranke@chromium.org> Commit-Queue: John Budorick <jbudorick@chromium.org> Cr-Commit-Position: refs/heads/master@{#537951} [modify] https://crrev.com/cdd30e47bed53573a91d9fb42563f3a5d4677484/testing/buildbot/README.md
Comment 1 by jbudorick@chromium.org
, Jan 11 2018