Recipes Build Tester needs to also evaluate recipe.cfg rolls |
|||
Issue descriptionCurrently the RBT evaluates the CL diff against 'recipes.py analyze'; This takes the list of touched files and lists which modules/recipes depend on them. It then sees if any of the chromium modules/recipes are in that list of transitive dependencies. If so, it then launches a led task to integration test the chromium recipe. However, this analysis misses the potential impact of upstream modules or the recipe engine changing their behavior (which would have stopped CIT PM #112 dead in its tracks). My proposal is: * Prepare a dummy CL which touches `network_service_content_browsertests` (a relatively fast, swarmed gtest test) * If recipes.cfg is in the changelist AND `recipes.py analyze` returns null (that is; just a roll without any recipe changes in the build repo itself) * Activate the led job and apply the dummy CL (currently the builder would just `pass` with no tryjob) The theory is that: * Some underlying infrastructure changed (engine, engine modules, depot_tools modules). * We DO use these things in the chromium recipe * But just running one swarmed test should be sufficient to exercise all these dependencies (bot_update, gclient, recipe_engine, swarming, isolate clients, cipd, etc.) but that running all the tests will be wasteful (time, resources) and will not add more information to the conclusion.
,
Jan 10
sgtm -- (following TPMs' recommendation for if owned, assert !untriaged)
,
Jan 11
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/f4a95b7b729a10a4bd4a645b14f04be02b01854f commit f4a95b7b729a10a4bd4a645b14f04be02b01854f Author: Robert Iannucci <iannucci@chromium.org> Date: Fri Jan 11 01:18:26 2019 [recipes/led_recipes_tester] Add recipe roll support. This allows led_recipes_tester to trigger an alternate (presumably faster) test CL for when it detects a recipes.cfg change. Recipes work inside the build repo which affect the chromium recipes should continue to test with the current (full) CL. R=martiniss@chromium.org, tandrii@chromium.org Bug: 920772 Change-Id: I8762e23e9ffa38df8d9bdae9104094ed4bc7fcbc Reviewed-on: https://chromium-review.googlesource.com/c/1405751 Commit-Queue: Robbie Iannucci <iannucci@chromium.org> Reviewed-by: Stephen Martinis <martiniss@chromium.org> Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org> [modify] https://crrev.com/f4a95b7b729a10a4bd4a645b14f04be02b01854f/scripts/slave/README.recipes.md [modify] https://crrev.com/f4a95b7b729a10a4bd4a645b14f04be02b01854f/scripts/slave/recipes/led_recipes_tester.expected/basic.json [modify] https://crrev.com/f4a95b7b729a10a4bd4a645b14f04be02b01854f/scripts/slave/recipes/led_recipes_tester.py [add] https://crrev.com/f4a95b7b729a10a4bd4a645b14f04be02b01854f/scripts/slave/recipes/led_recipes_tester.expected/recipe_roller.json [add] https://crrev.com/f4a95b7b729a10a4bd4a645b14f04be02b01854f/scripts/slave/recipes/led_recipes_tester.expected/manual_roll_with_changes.json
,
Jan 11
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/recipes-py/+/27cd207dfb18d832d94218e61720acbb0127a26b commit 27cd207dfb18d832d94218e61720acbb0127a26b Author: Robert Iannucci <iannucci@chromium.org> Date: Fri Jan 11 02:26:44 2019 Whitespace change to watch Build Recipes Tester. TBR=jchinlee@chromium.org, martiniss@chromium.org, tandrii@chromium.org Bug: 920772 Change-Id: I65a75881c1f8a04cb55af52b223ddb9a96306ed9 Reviewed-on: https://chromium-review.googlesource.com/c/1405516 Commit-Queue: Robbie Iannucci <iannucci@chromium.org> Reviewed-by: Robbie Iannucci <iannucci@chromium.org> [modify] https://crrev.com/27cd207dfb18d832d94218e61720acbb0127a26b/recipe_engine/doc.py
,
Jan 11
Here it is, checking a recipe roll: * Roll: https://chromium-review.googlesource.com/c/chromium/tools/build/+/1405627/ * Tryjob: https://ci.chromium.org/p/infra/builders/luci.infra.try/Build%20Recipes%20Tester/b8924663830793774096 Tryjob takes 14 minutes, which seems quite fair (and it does successfully run the interactive ui tests (which I picked over `network_service_content_browsertests` once I understood how the "network_service_*" test variants worked). Calling this one fixed :)
,
Jan 11
Good job! |
|||
►
Sign in to add a comment |
|||
Comment 1 by iannu...@google.com
, Jan 10