Make testing/buildbot specs fully specify a test's swarming dimensions |
||
Issue descriptionIn other words, don't let the chromium recipe apply defaults. This would be nice because it's hard to tell from the testing specs alone what a given test will run on. Take this bot's tests for instance: https://chromium.googlesource.com/chromium/src/+/5e05592b7539d0dfae6e25cd69db02791f97c226/testing/buildbot/chromium.clang.json#12934 What OS will it run on? Mac presumably... but what version of Mac? And what cpu arch? What swarming pool will it run in? Will it want a gpu? Here's a task that the bot launches: https://chromium-swarm.appspot.com/task?id=3fc0d48168af5e10 It requests 4 separate dimensions. We should specify all those in the testing specs, and not let the recipe add them. Here's all the places (that I'm aware of) where it currently does that: - from a builder's recipe_module/chromium_tests definition: https://codesearch.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/chromium_mac.py?rcl=d01e16520310e96cdd8f29e9032ebf7245ef29fd&l=125 - from the chromium_swarm recipe_module: https://codesearch.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_swarming/api.py?rcl=d01e16520310e96cdd8f29e9032ebf7245ef29fd&l=11 https://codesearch.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_swarming/api.py?rcl=d01e16520310e96cdd8f29e9032ebf7245ef29fd&l=54 - from the swarming recipe_module: https://codesearch.chromium.org/chromium/build/scripts/slave/recipe_modules/swarming/api.py?rcl=d01e16520310e96cdd8f29e9032ebf7245ef29fd&l=186 This would likely require adding most of that default-applying logic to generate_buildbot_json.py (which, IMO, is where it should live). +some folks that might have opinions here.
,
Sep 5
Happy to work on this at some point as well.
,
Sep 5
I support this idea too and am happy to help implement it. One thing which would be helpful in generate_buildbot_json.py (and in particular waterfalls.pyl) would be the ability to name certain groups of Swarming dimensions so that machines can refer to them easily, rather than replicating them.
,
Sep 5
MA added the swarming recipe module default in https://crrev.com/d53fae45108858c98e35c1abe2aa8ee03c5ba9ff (back in 2014). I think that should stay? It seems like a sane default to have. The others should probably move source side. I agree with #3 about having some sort of indirection or something. WDYT about doing something like gatekeeper, which allows you to specify a setting at a master which gets applied to every builder automatically. Although that would only really work for the chromium_tests definition, not for the PER_TARGET_SWARMING_DIMS stuff.
,
Sep 5
Also, I think I'm going to work on this, as it's very related to my work in bug 868153 and bug 867054
,
Sep 5
#4: I don't think a per-master specification applied automatically will work. Waterfalls like: https://ci.chromium.org/p/chromium/g/chromium.webkit/console and https://ci.chromium.org/p/chromium/g/chromium.gpu.fyi/console have a large number of differently configured bots on them. On the other hand, bots with the same Swarming dimensions may show up on other waterfalls, too. I've been thinking about specifying one big dictionary at the top of waterfalls.pyl like "named_swarming_dimension_sets" which would contain key/value pairs which could be referred to by bots later. So for example: [ { 'named_swarming_dimension_sets': { 'WIN10_NVIDIA_QUADRO_P400_STABLE_DRIVER': { 'gpu': '10de:1cb3-23.21.13.8816', 'os': 'Windows-10', 'pool': 'Chrome-GPU', } } } // Later in waterfalls.pyl... { 'name': 'chromium.gpu', 'machines': { 'Win10 Debug (NVIDIA)': { 'swarming': { 'dimension_sets': [ { 'name': 'WIN10_NVIDIA_QUADRO_P400_STABLE_DRIVER', }, ], }, // ... }, 'Win10 Release (NVIDIA)': { 'swarming': { 'dimension_sets': [ { 'name': 'WIN10_NVIDIA_QUADRO_P400_STABLE_DRIVER', }, ], }, // ... }, } } ] Need to think through how to specialize them further per-bot. What do you think about that general possibility?
,
Sep 5
I think that's a good idea. But maybe call it mixins, similar to what's in mb_config.pyl. Could even put the mixins in another file too. I think that'd be pretty flexible.
,
Sep 5
So for the example, you'd just have
{
..
'Win10 Release (NVIDIA)': {
'swarming': {
'dimension_sets': [{
'mixins': ['win10', 'gpu', 'nvidia_quadro_p400_stable'],
}],
},
}
..
}
,
Sep 6
Sounds great.
,
Sep 6
SGTM also.
,
Sep 13
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/b6a5049b4d44fa1ebbfc8fb75709be00dbcca776 commit b6a5049b4d44fa1ebbfc8fb75709be00dbcca776 Author: Stephen Martinis <martiniss@chromium.org> Date: Wed Sep 12 23:59:32 2018 generate_buildbot_json.py: Add swarming mixins This CL adds swarming_mixins to generate_buildbot_json.py. This will allow us to move most of the dimension setting that currently occurs in the recipes to the generation scripts here, which should help to make sure you only need to look in one place to see what tests are being run with what dimensions. Bug: 880973 Change-Id: I59e0aac1f017c785c14a98f574ff638f7064c9dd Reviewed-on: https://chromium-review.googlesource.com/1214427 Commit-Queue: Stephen Martinis <martiniss@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#590865} [modify] https://crrev.com/b6a5049b4d44fa1ebbfc8fb75709be00dbcca776/testing/buildbot/generate_buildbot_json.py [modify] https://crrev.com/b6a5049b4d44fa1ebbfc8fb75709be00dbcca776/testing/buildbot/generate_buildbot_json_unittest.py [add] https://crrev.com/b6a5049b4d44fa1ebbfc8fb75709be00dbcca776/testing/buildbot/swarming_mixins.pyl
,
Sep 14
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/9fc367df570d7876a400fce21aaa26b2c7625a92 commit 9fc367df570d7876a400fce21aaa26b2c7625a92 Author: Stephen Martinis <martiniss@chromium.org> Date: Fri Sep 14 00:54:35 2018 Linux ARM: Use swarming mixins This change is deliberately small (only changing one builder) to make sure the feature is working correctly. Bug: 880973 Change-Id: If9ffea3eadc461a994cd7dcc409b9b8206871024 Reviewed-on: https://chromium-review.googlesource.com/1222165 Reviewed-by: Kenneth Russell <kbr@chromium.org> Commit-Queue: Stephen Martinis <martiniss@chromium.org> Cr-Commit-Position: refs/heads/master@{#591241} [modify] https://crrev.com/9fc367df570d7876a400fce21aaa26b2c7625a92/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/9fc367df570d7876a400fce21aaa26b2c7625a92/testing/buildbot/swarming_mixins.pyl [modify] https://crrev.com/9fc367df570d7876a400fce21aaa26b2c7625a92/testing/buildbot/waterfalls.pyl
,
Sep 14
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/260972ddb5e56ba931033b239a769dbb94f32b20 commit 260972ddb5e56ba931033b239a769dbb94f32b20 Author: Stephen Martinis <martiniss@chromium.org> Date: Fri Sep 14 21:40:49 2018 Revert "Linux ARM: Use swarming mixins" This reverts commit 9fc367df570d7876a400fce21aaa26b2c7625a92. Reason for revert: Broke bot Original change's description: > Linux ARM: Use swarming mixins > > This change is deliberately small (only changing one builder) to make > sure the feature is working correctly. > > Bug: 880973 > Change-Id: If9ffea3eadc461a994cd7dcc409b9b8206871024 > Reviewed-on: https://chromium-review.googlesource.com/1222165 > Reviewed-by: Kenneth Russell <kbr@chromium.org> > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > Cr-Commit-Position: refs/heads/master@{#591241} TBR=dpranke@chromium.org,kbr@chromium.org,martiniss@chromium.org,bpastene@chromium.org,jbudorick@chromium.org Change-Id: I0aca3b5b53c5f74f0af8f5fc287051cc4457cdd2 No-Presubmit: true No-Tree-Checks: true No-Try: true Bug: 880973 Reviewed-on: https://chromium-review.googlesource.com/1227394 Reviewed-by: Stephen Martinis <martiniss@chromium.org> Commit-Queue: Stephen Martinis <martiniss@chromium.org> Cr-Commit-Position: refs/heads/master@{#591484} [modify] https://crrev.com/260972ddb5e56ba931033b239a769dbb94f32b20/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/260972ddb5e56ba931033b239a769dbb94f32b20/testing/buildbot/swarming_mixins.pyl [modify] https://crrev.com/260972ddb5e56ba931033b239a769dbb94f32b20/testing/buildbot/waterfalls.pyl
,
Sep 15
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/089b183e549db3e996b063d2a43c3b62197e2552 commit 089b183e549db3e996b063d2a43c3b62197e2552 Author: Stephen Martinis <martiniss@chromium.org> Date: Sat Sep 15 01:21:07 2018 Reland "Linux ARM: Use swarming mixins" This is a reland of 9fc367df570d7876a400fce21aaa26b2c7625a92 Original change's description: > Linux ARM: Use swarming mixins > > This change is deliberately small (only changing one builder) to make > sure the feature is working correctly. > > Bug: 880973 > Change-Id: If9ffea3eadc461a994cd7dcc409b9b8206871024 > Reviewed-on: https://chromium-review.googlesource.com/1222165 > Reviewed-by: Kenneth Russell <kbr@chromium.org> > Commit-Queue: Stephen Martinis <martiniss@chromium.org> > Cr-Commit-Position: refs/heads/master@{#591241} Bug: 880973 Change-Id: I8a5b965cc470caf660b9ade9ba12af7bc2d92a12 Reviewed-on: https://chromium-review.googlesource.com/1227396 Reviewed-by: Ben Pastene <bpastene@chromium.org> Commit-Queue: Stephen Martinis <martiniss@chromium.org> Cr-Commit-Position: refs/heads/master@{#591550} [modify] https://crrev.com/089b183e549db3e996b063d2a43c3b62197e2552/testing/buildbot/chromium.fyi.json [modify] https://crrev.com/089b183e549db3e996b063d2a43c3b62197e2552/testing/buildbot/swarming_mixins.pyl [modify] https://crrev.com/089b183e549db3e996b063d2a43c3b62197e2552/testing/buildbot/waterfalls.pyl
,
Sep 17
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/0382bc181ae1791a87705d84688f1a6bd17df4bc commit 0382bc181ae1791a87705d84688f1a6bd17df4bc Author: Stephen Martinis <martiniss@chromium.org> Date: Mon Sep 17 22:29:07 2018 generate_buildbot_json: Modify swarming_mixins Now applies mixins before modifications, so modifications always take precedent. Also: * Applies the swarming mixins only to the 'swarming' key in a test. * Adds a test to ensure all mixins are used. Bug: 880973 Change-Id: I6289bb32cd331fab2d5a182f0b31cee050d2cabf Reviewed-on: https://chromium-review.googlesource.com/1227408 Commit-Queue: Stephen Martinis <martiniss@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#591845} [modify] https://crrev.com/0382bc181ae1791a87705d84688f1a6bd17df4bc/testing/buildbot/generate_buildbot_json.py [modify] https://crrev.com/0382bc181ae1791a87705d84688f1a6bd17df4bc/testing/buildbot/generate_buildbot_json_unittest.py
,
Sep 18
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/617c7dfd63b56241642e7f85f122726a1be8c7dd commit 617c7dfd63b56241642e7f85f122726a1be8c7dd Author: Stephen Martinis <martiniss@chromium.org> Date: Tue Sep 18 22:55:31 2018 Linux ARM: Remove swarming dimensions These are now set source side in https://crrev.com/c/1222165, so we don't need them in the recipes. Bug: 880973 Change-Id: I7225974ebbabb1cc9fd6a043c6e177f8e579494f Reviewed-on: https://chromium-review.googlesource.com/1226084 Auto-Submit: Stephen Martinis <martiniss@chromium.org> Reviewed-by: Ben Pastene <bpastene@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> [modify] https://crrev.com/617c7dfd63b56241642e7f85f122726a1be8c7dd/scripts/slave/recipe_modules/chromium_tests/chromium_fyi.py
,
Sep 20
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/f5f4ea27321c185aa18ff751734bb8777b4d093c commit f5f4ea27321c185aa18ff751734bb8777b4d093c Author: Stephen Martinis <martiniss@chromium.org> Date: Thu Sep 20 01:07:43 2018 //testing/buildbot: Use mixins for chromium.mac This CL enforces swarming dimensions for the mac testers on chromium.mac using swarming_mixins. Bug: 880973 Change-Id: I3982de0c6704ceaa2f8e3ab5b9909c5c01319aa6 Reviewed-on: https://chromium-review.googlesource.com/1223027 Commit-Queue: Stephen Martinis <martiniss@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Reviewed-by: Ben Pastene <bpastene@chromium.org> Reviewed-by: Kenneth Russell <kbr@chromium.org> Cr-Commit-Position: refs/heads/master@{#592646} [modify] https://crrev.com/f5f4ea27321c185aa18ff751734bb8777b4d093c/testing/buildbot/chromium.mac.json [modify] https://crrev.com/f5f4ea27321c185aa18ff751734bb8777b4d093c/testing/buildbot/generate_buildbot_json.py [modify] https://crrev.com/f5f4ea27321c185aa18ff751734bb8777b4d093c/testing/buildbot/swarming_mixins.pyl [modify] https://crrev.com/f5f4ea27321c185aa18ff751734bb8777b4d093c/testing/buildbot/waterfalls.pyl
,
Sep 21
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/e11f1f639e8582373bda963fd73633bee93d7dc2 commit e11f1f639e8582373bda963fd73633bee93d7dc2 Author: Stephen Martinis <martiniss@chromium.org> Date: Fri Sep 21 23:42:55 2018 Remove mac swarming dimensions These were moved to source in https://crrev.com/c/1223027 Bug: 880973 Change-Id: Ie7e02b3342a4b7ea01b8e4298da62bb3d74df92a Reviewed-on: https://chromium-review.googlesource.com/1239594 Reviewed-by: John Budorick <jbudorick@chromium.org> Commit-Queue: Stephen Martinis <martiniss@chromium.org> [modify] https://crrev.com/e11f1f639e8582373bda963fd73633bee93d7dc2/scripts/slave/recipe_modules/chromium_tests/chromium_mac.py
,
Sep 24
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/3284554419d1a86a42942db41f81d022a63cba09 commit 3284554419d1a86a42942db41f81d022a63cba09 Author: Stephen Martinis <martiniss@chromium.org> Date: Mon Sep 24 19:38:28 2018 Add chromium.win swarming dimensions Bug: 880973 Change-Id: I905435107c3568d6e9f6c8eaf4c5fc9ec1173746 Reviewed-on: https://chromium-review.googlesource.com/1239622 Commit-Queue: Stephen Martinis <martiniss@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Cr-Commit-Position: refs/heads/master@{#593636} [modify] https://crrev.com/3284554419d1a86a42942db41f81d022a63cba09/testing/buildbot/chromium.win.json [modify] https://crrev.com/3284554419d1a86a42942db41f81d022a63cba09/testing/buildbot/swarming_mixins.pyl [modify] https://crrev.com/3284554419d1a86a42942db41f81d022a63cba09/testing/buildbot/waterfalls.pyl
,
Sep 24
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/4e82faa21897c2b6e2e6dfcdf8ae6727abf0fb71 commit 4e82faa21897c2b6e2e6dfcdf8ae6727abf0fb71 Author: Stephen Martinis <martiniss@chromium.org> Date: Mon Sep 24 22:01:08 2018 Add chromium.memory swarming dimensions Bug: 880973 Change-Id: Ic92339a48740e4e18781d81861252c9fc6424d2a Reviewed-on: https://chromium-review.googlesource.com/1239629 Commit-Queue: Stephen Martinis <martiniss@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Cr-Commit-Position: refs/heads/master@{#593704} [modify] https://crrev.com/4e82faa21897c2b6e2e6dfcdf8ae6727abf0fb71/testing/buildbot/chromium.memory.json [modify] https://crrev.com/4e82faa21897c2b6e2e6dfcdf8ae6727abf0fb71/testing/buildbot/waterfalls.pyl
,
Sep 25
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/1062bb719adb9de570f438e8bf34b68a7279e89d commit 1062bb719adb9de570f438e8bf34b68a7279e89d Author: Stephen Martinis <martiniss@chromium.org> Date: Tue Sep 25 23:46:08 2018 Remove windows swarming dimensions These are specified in https://crrev.com/c/1239622 Bug: 880973 Change-Id: I9dd6791539e2a2c17e91f5cfe9539926c40e643b Reviewed-on: https://chromium-review.googlesource.com/1239595 Reviewed-by: Ben Pastene <bpastene@chromium.org> Commit-Queue: Stephen Martinis <martiniss@chromium.org> [modify] https://crrev.com/1062bb719adb9de570f438e8bf34b68a7279e89d/scripts/slave/recipe_modules/chromium_tests/chromium_win.py
,
Sep 25
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/9eb591f4fb296b7217a0ec1ec2e595d95e263ece commit 9eb591f4fb296b7217a0ec1ec2e595d95e263ece Author: Stephen Martinis <martiniss@chromium.org> Date: Tue Sep 25 23:48:40 2018 Remove swarming_dimensions for chromium.memory These were migrated source side in https://crrev.com/c/1239629 Bug: 880973 Change-Id: Iccb2654fdab219464ed5779cee1841472ed0f082 Reviewed-on: https://chromium-review.googlesource.com/1244128 Auto-Submit: Stephen Martinis <martiniss@chromium.org> Reviewed-by: Ben Pastene <bpastene@chromium.org> Commit-Queue: Ben Pastene <bpastene@chromium.org> [modify] https://crrev.com/9eb591f4fb296b7217a0ec1ec2e595d95e263ece/scripts/slave/recipe_modules/chromium_tests/chromium_memory.py
,
Sep 28
The following revision refers to this bug: https://chromium.googlesource.com/chromium/src.git/+/33da7cd41e0301772ff68b83bb72840bdddbb002 commit 33da7cd41e0301772ff68b83bb72840bdddbb002 Author: Stephen Martinis <martiniss@chromium.org> Date: Fri Sep 28 00:11:35 2018 //testing/buildbot: Mixins for webrtc experimental Use swarming mixins for webrtc experimental bots. Bug: 880973 Change-Id: I51e386a78f8616a94c9c8a0d67de5bdf096dfe2b Reviewed-on: https://chromium-review.googlesource.com/1246946 Commit-Queue: Stephen Martinis <martiniss@chromium.org> Reviewed-by: John Budorick <jbudorick@chromium.org> Cr-Commit-Position: refs/heads/master@{#594925} [modify] https://crrev.com/33da7cd41e0301772ff68b83bb72840bdddbb002/testing/buildbot/chromium.webrtc.fyi.experimental.json [modify] https://crrev.com/33da7cd41e0301772ff68b83bb72840bdddbb002/testing/buildbot/swarming_mixins.pyl [modify] https://crrev.com/33da7cd41e0301772ff68b83bb72840bdddbb002/testing/buildbot/waterfalls.pyl
,
Sep 28
The following revision refers to this bug: https://chromium.googlesource.com/chromium/tools/build/+/0094aab98c5fba78d3efbe887a23fd93c6f00ce2 commit 0094aab98c5fba78d3efbe887a23fd93c6f00ce2 Author: Stephen Martinis <martiniss@chromium.org> Date: Fri Sep 28 07:39:17 2018 Remove swarming_dimensions for webrtc experimental bots These are implemented source side in https://crrev.com/c/1246946. Bug: 880973 Change-Id: I9602904467d730e70766efc970d9c8a6cd26b6ea Reviewed-on: https://chromium-review.googlesource.com/1246480 Reviewed-by: Garrett Beaty <gbeaty@chromium.org> Commit-Queue: Patrik Höglund <phoglund@chromium.org> [modify] https://crrev.com/0094aab98c5fba78d3efbe887a23fd93c6f00ce2/scripts/slave/recipe_modules/chromium_tests/chromium_webrtc_fyi_experimental.py
,
Oct 9
The following revision refers to this bug: https://chrome-internal.googlesource.com/clank/internal/apps/+/4de9c50d1ae10da2a7036aaba3bc78d1c71742ed commit 4de9c50d1ae10da2a7036aaba3bc78d1c71742ed Author: Stephen Martinis <martiniss@google.com> Date: Tue Oct 09 18:16:30 2018 |
||
►
Sign in to add a comment |
||
Comment 1 by martiniss@chromium.org
, Sep 5