New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 880973 link

Starred by 2 users

Issue metadata

Status: Assigned
Owner:
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug



Sign in to add a comment

Make testing/buildbot specs fully specify a test's swarming dimensions

Project Member Reported by bpastene@chromium.org, Sep 5

Issue description

In other words, don't let the chromium recipe apply defaults. This would be nice because it's hard to tell from the testing specs alone what a given test will run on. Take this bot's tests for instance:
https://chromium.googlesource.com/chromium/src/+/5e05592b7539d0dfae6e25cd69db02791f97c226/testing/buildbot/chromium.clang.json#12934
What OS will it run on? Mac presumably... but what version of Mac? And what cpu arch? What swarming pool will it run in? Will it want a gpu?

Here's a task that the bot launches:
https://chromium-swarm.appspot.com/task?id=3fc0d48168af5e10

It requests 4 separate dimensions. We should specify all those in the testing specs, and not let the recipe add them. Here's all the places (that I'm aware of) where it currently does that:

- from a builder's recipe_module/chromium_tests definition:
https://codesearch.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_tests/chromium_mac.py?rcl=d01e16520310e96cdd8f29e9032ebf7245ef29fd&l=125

- from the chromium_swarm recipe_module:
https://codesearch.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_swarming/api.py?rcl=d01e16520310e96cdd8f29e9032ebf7245ef29fd&l=11
https://codesearch.chromium.org/chromium/build/scripts/slave/recipe_modules/chromium_swarming/api.py?rcl=d01e16520310e96cdd8f29e9032ebf7245ef29fd&l=54

- from the swarming recipe_module:
https://codesearch.chromium.org/chromium/build/scripts/slave/recipe_modules/swarming/api.py?rcl=d01e16520310e96cdd8f29e9032ebf7245ef29fd&l=186

This would likely require adding most of that default-applying logic to generate_buildbot_json.py (which, IMO, is where it should live). +some folks that might have opinions here.
 
I am in full support of this.
Happy to work on this at some point as well.
I support this idea too and am happy to help implement it. One thing which would be helpful in generate_buildbot_json.py (and in particular waterfalls.pyl) would be the ability to name certain groups of Swarming dimensions so that machines can refer to them easily, rather than replicating them.

MA added the swarming recipe module default in https://crrev.com/d53fae45108858c98e35c1abe2aa8ee03c5ba9ff (back in 2014). I think that should stay? It seems like a sane default to have.

The others should probably move source side.

I agree with #3 about having some sort of indirection or something. WDYT about doing something like gatekeeper, which allows you to specify a setting at a master which gets applied to every builder automatically. Although that would only really work for the chromium_tests definition, not for the PER_TARGET_SWARMING_DIMS stuff.
Owner: martiniss@chromium.org
Status: Assigned (was: Available)
Also, I think I'm going to work on this, as it's very related to my work in bug 868153 and bug 867054
#4: I don't think a per-master specification applied automatically will work. Waterfalls like:

https://ci.chromium.org/p/chromium/g/chromium.webkit/console

and

https://ci.chromium.org/p/chromium/g/chromium.gpu.fyi/console

have a large number of differently configured bots on them. On the other hand, bots with the same Swarming dimensions may show up on other waterfalls, too.

I've been thinking about specifying one big dictionary at the top of waterfalls.pyl like "named_swarming_dimension_sets" which would contain key/value pairs which could be referred to by bots later. So for example:

[
  {
     'named_swarming_dimension_sets': {
       'WIN10_NVIDIA_QUADRO_P400_STABLE_DRIVER': {
         'gpu': '10de:1cb3-23.21.13.8816',
         'os': 'Windows-10',
         'pool': 'Chrome-GPU',
       }
     }
  }
  // Later in waterfalls.pyl...
  {
    'name': 'chromium.gpu',
    'machines': {
      'Win10 Debug (NVIDIA)': {
        'swarming': {
          'dimension_sets': [
            {
              'name': 'WIN10_NVIDIA_QUADRO_P400_STABLE_DRIVER',
             },
          ],
        },
        // ...
      },
      'Win10 Release (NVIDIA)': {
        'swarming': {
          'dimension_sets': [
            {
              'name': 'WIN10_NVIDIA_QUADRO_P400_STABLE_DRIVER',
             },
          ],
        },
        // ...
      },
    }
  }
]

Need to think through how to specialize them further per-bot.

What do you think about that general possibility?

I think that's a good idea. But maybe call it mixins, similar to what's in mb_config.pyl. Could even put the mixins in another file too. I think that'd be pretty flexible.
So for the example, you'd just have

{
..
'Win10 Release (NVIDIA)': {
  'swarming': {
     'dimension_sets': [{
        'mixins': ['win10', 'gpu', 'nvidia_quadro_p400_stable'],
      }],
    },
}
..
}
Sounds great.

SGTM also.
Project Member

Comment 11 by bugdroid1@chromium.org, Sep 13

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b6a5049b4d44fa1ebbfc8fb75709be00dbcca776

commit b6a5049b4d44fa1ebbfc8fb75709be00dbcca776
Author: Stephen Martinis <martiniss@chromium.org>
Date: Wed Sep 12 23:59:32 2018

generate_buildbot_json.py: Add swarming mixins

This CL adds swarming_mixins to generate_buildbot_json.py. This will
allow us to move most of the dimension setting that currently occurs
in the recipes to the generation scripts here, which should help to make
sure you only need to look in one place to see what tests are being run
with what dimensions.

Bug: 880973
Change-Id: I59e0aac1f017c785c14a98f574ff638f7064c9dd
Reviewed-on: https://chromium-review.googlesource.com/1214427
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#590865}
[modify] https://crrev.com/b6a5049b4d44fa1ebbfc8fb75709be00dbcca776/testing/buildbot/generate_buildbot_json.py
[modify] https://crrev.com/b6a5049b4d44fa1ebbfc8fb75709be00dbcca776/testing/buildbot/generate_buildbot_json_unittest.py
[add] https://crrev.com/b6a5049b4d44fa1ebbfc8fb75709be00dbcca776/testing/buildbot/swarming_mixins.pyl

Project Member

Comment 12 by bugdroid1@chromium.org, Sep 14

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/9fc367df570d7876a400fce21aaa26b2c7625a92

commit 9fc367df570d7876a400fce21aaa26b2c7625a92
Author: Stephen Martinis <martiniss@chromium.org>
Date: Fri Sep 14 00:54:35 2018

Linux ARM: Use swarming mixins

This change is deliberately small (only changing one builder) to make
sure the feature is working correctly.

Bug: 880973
Change-Id: If9ffea3eadc461a994cd7dcc409b9b8206871024
Reviewed-on: https://chromium-review.googlesource.com/1222165
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#591241}
[modify] https://crrev.com/9fc367df570d7876a400fce21aaa26b2c7625a92/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/9fc367df570d7876a400fce21aaa26b2c7625a92/testing/buildbot/swarming_mixins.pyl
[modify] https://crrev.com/9fc367df570d7876a400fce21aaa26b2c7625a92/testing/buildbot/waterfalls.pyl

Project Member

Comment 13 by bugdroid1@chromium.org, Sep 14

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/260972ddb5e56ba931033b239a769dbb94f32b20

commit 260972ddb5e56ba931033b239a769dbb94f32b20
Author: Stephen Martinis <martiniss@chromium.org>
Date: Fri Sep 14 21:40:49 2018

Revert "Linux ARM: Use swarming mixins"

This reverts commit 9fc367df570d7876a400fce21aaa26b2c7625a92.

Reason for revert: Broke bot

Original change's description:
> Linux ARM: Use swarming mixins
> 
> This change is deliberately small (only changing one builder) to make
> sure the feature is working correctly.
> 
> Bug: 880973
> Change-Id: If9ffea3eadc461a994cd7dcc409b9b8206871024
> Reviewed-on: https://chromium-review.googlesource.com/1222165
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#591241}

TBR=dpranke@chromium.org,kbr@chromium.org,martiniss@chromium.org,bpastene@chromium.org,jbudorick@chromium.org

Change-Id: I0aca3b5b53c5f74f0af8f5fc287051cc4457cdd2
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: 880973
Reviewed-on: https://chromium-review.googlesource.com/1227394
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#591484}
[modify] https://crrev.com/260972ddb5e56ba931033b239a769dbb94f32b20/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/260972ddb5e56ba931033b239a769dbb94f32b20/testing/buildbot/swarming_mixins.pyl
[modify] https://crrev.com/260972ddb5e56ba931033b239a769dbb94f32b20/testing/buildbot/waterfalls.pyl

Project Member

Comment 14 by bugdroid1@chromium.org, Sep 15

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/089b183e549db3e996b063d2a43c3b62197e2552

commit 089b183e549db3e996b063d2a43c3b62197e2552
Author: Stephen Martinis <martiniss@chromium.org>
Date: Sat Sep 15 01:21:07 2018

Reland "Linux ARM: Use swarming mixins"

This is a reland of 9fc367df570d7876a400fce21aaa26b2c7625a92

Original change's description:
> Linux ARM: Use swarming mixins
> 
> This change is deliberately small (only changing one builder) to make
> sure the feature is working correctly.
> 
> Bug: 880973
> Change-Id: If9ffea3eadc461a994cd7dcc409b9b8206871024
> Reviewed-on: https://chromium-review.googlesource.com/1222165
> Reviewed-by: Kenneth Russell <kbr@chromium.org>
> Commit-Queue: Stephen Martinis <martiniss@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#591241}

Bug: 880973
Change-Id: I8a5b965cc470caf660b9ade9ba12af7bc2d92a12
Reviewed-on: https://chromium-review.googlesource.com/1227396
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Cr-Commit-Position: refs/heads/master@{#591550}
[modify] https://crrev.com/089b183e549db3e996b063d2a43c3b62197e2552/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/089b183e549db3e996b063d2a43c3b62197e2552/testing/buildbot/swarming_mixins.pyl
[modify] https://crrev.com/089b183e549db3e996b063d2a43c3b62197e2552/testing/buildbot/waterfalls.pyl

Project Member

Comment 15 by bugdroid1@chromium.org, Sep 17

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0382bc181ae1791a87705d84688f1a6bd17df4bc

commit 0382bc181ae1791a87705d84688f1a6bd17df4bc
Author: Stephen Martinis <martiniss@chromium.org>
Date: Mon Sep 17 22:29:07 2018

generate_buildbot_json: Modify swarming_mixins

Now applies mixins before modifications, so modifications always take
precedent.

Also:
* Applies the swarming mixins only to the 'swarming' key in a test.
* Adds a test to ensure all mixins are used.

Bug: 880973
Change-Id: I6289bb32cd331fab2d5a182f0b31cee050d2cabf
Reviewed-on: https://chromium-review.googlesource.com/1227408
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#591845}
[modify] https://crrev.com/0382bc181ae1791a87705d84688f1a6bd17df4bc/testing/buildbot/generate_buildbot_json.py
[modify] https://crrev.com/0382bc181ae1791a87705d84688f1a6bd17df4bc/testing/buildbot/generate_buildbot_json_unittest.py

Project Member

Comment 16 by bugdroid1@chromium.org, Sep 18

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/617c7dfd63b56241642e7f85f122726a1be8c7dd

commit 617c7dfd63b56241642e7f85f122726a1be8c7dd
Author: Stephen Martinis <martiniss@chromium.org>
Date: Tue Sep 18 22:55:31 2018

Linux ARM: Remove swarming dimensions

These are now set source side in https://crrev.com/c/1222165,
so we don't need them in the recipes.

Bug: 880973
Change-Id: I7225974ebbabb1cc9fd6a043c6e177f8e579494f
Reviewed-on: https://chromium-review.googlesource.com/1226084
Auto-Submit: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>

[modify] https://crrev.com/617c7dfd63b56241642e7f85f122726a1be8c7dd/scripts/slave/recipe_modules/chromium_tests/chromium_fyi.py

Project Member

Comment 17 by bugdroid1@chromium.org, Sep 20

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f5f4ea27321c185aa18ff751734bb8777b4d093c

commit f5f4ea27321c185aa18ff751734bb8777b4d093c
Author: Stephen Martinis <martiniss@chromium.org>
Date: Thu Sep 20 01:07:43 2018

//testing/buildbot: Use mixins for chromium.mac

This CL enforces swarming dimensions for the mac testers on
chromium.mac using swarming_mixins.

Bug: 880973
Change-Id: I3982de0c6704ceaa2f8e3ab5b9909c5c01319aa6
Reviewed-on: https://chromium-review.googlesource.com/1223027
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Reviewed-by: Kenneth Russell <kbr@chromium.org>
Cr-Commit-Position: refs/heads/master@{#592646}
[modify] https://crrev.com/f5f4ea27321c185aa18ff751734bb8777b4d093c/testing/buildbot/chromium.mac.json
[modify] https://crrev.com/f5f4ea27321c185aa18ff751734bb8777b4d093c/testing/buildbot/generate_buildbot_json.py
[modify] https://crrev.com/f5f4ea27321c185aa18ff751734bb8777b4d093c/testing/buildbot/swarming_mixins.pyl
[modify] https://crrev.com/f5f4ea27321c185aa18ff751734bb8777b4d093c/testing/buildbot/waterfalls.pyl

Project Member

Comment 18 by bugdroid1@chromium.org, Sep 21

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/e11f1f639e8582373bda963fd73633bee93d7dc2

commit e11f1f639e8582373bda963fd73633bee93d7dc2
Author: Stephen Martinis <martiniss@chromium.org>
Date: Fri Sep 21 23:42:55 2018

Remove mac swarming dimensions

These were moved to source in https://crrev.com/c/1223027

Bug: 880973
Change-Id: Ie7e02b3342a4b7ea01b8e4298da62bb3d74df92a
Reviewed-on: https://chromium-review.googlesource.com/1239594
Reviewed-by: John Budorick <jbudorick@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>

[modify] https://crrev.com/e11f1f639e8582373bda963fd73633bee93d7dc2/scripts/slave/recipe_modules/chromium_tests/chromium_mac.py

Project Member

Comment 19 by bugdroid1@chromium.org, Sep 24

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/3284554419d1a86a42942db41f81d022a63cba09

commit 3284554419d1a86a42942db41f81d022a63cba09
Author: Stephen Martinis <martiniss@chromium.org>
Date: Mon Sep 24 19:38:28 2018

Add chromium.win swarming dimensions

Bug: 880973
Change-Id: I905435107c3568d6e9f6c8eaf4c5fc9ec1173746
Reviewed-on: https://chromium-review.googlesource.com/1239622
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#593636}
[modify] https://crrev.com/3284554419d1a86a42942db41f81d022a63cba09/testing/buildbot/chromium.win.json
[modify] https://crrev.com/3284554419d1a86a42942db41f81d022a63cba09/testing/buildbot/swarming_mixins.pyl
[modify] https://crrev.com/3284554419d1a86a42942db41f81d022a63cba09/testing/buildbot/waterfalls.pyl

Project Member

Comment 20 by bugdroid1@chromium.org, Sep 24

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/4e82faa21897c2b6e2e6dfcdf8ae6727abf0fb71

commit 4e82faa21897c2b6e2e6dfcdf8ae6727abf0fb71
Author: Stephen Martinis <martiniss@chromium.org>
Date: Mon Sep 24 22:01:08 2018

Add chromium.memory swarming dimensions

Bug: 880973
Change-Id: Ic92339a48740e4e18781d81861252c9fc6424d2a
Reviewed-on: https://chromium-review.googlesource.com/1239629
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#593704}
[modify] https://crrev.com/4e82faa21897c2b6e2e6dfcdf8ae6727abf0fb71/testing/buildbot/chromium.memory.json
[modify] https://crrev.com/4e82faa21897c2b6e2e6dfcdf8ae6727abf0fb71/testing/buildbot/waterfalls.pyl

Project Member

Comment 21 by bugdroid1@chromium.org, Sep 25

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/1062bb719adb9de570f438e8bf34b68a7279e89d

commit 1062bb719adb9de570f438e8bf34b68a7279e89d
Author: Stephen Martinis <martiniss@chromium.org>
Date: Tue Sep 25 23:46:08 2018

Remove windows swarming dimensions

These are specified in https://crrev.com/c/1239622

Bug: 880973
Change-Id: I9dd6791539e2a2c17e91f5cfe9539926c40e643b
Reviewed-on: https://chromium-review.googlesource.com/1239595
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Stephen Martinis <martiniss@chromium.org>

[modify] https://crrev.com/1062bb719adb9de570f438e8bf34b68a7279e89d/scripts/slave/recipe_modules/chromium_tests/chromium_win.py

Project Member

Comment 22 by bugdroid1@chromium.org, Sep 25

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/9eb591f4fb296b7217a0ec1ec2e595d95e263ece

commit 9eb591f4fb296b7217a0ec1ec2e595d95e263ece
Author: Stephen Martinis <martiniss@chromium.org>
Date: Tue Sep 25 23:48:40 2018

Remove swarming_dimensions for chromium.memory

These were migrated source side in https://crrev.com/c/1239629

Bug: 880973
Change-Id: Iccb2654fdab219464ed5779cee1841472ed0f082
Reviewed-on: https://chromium-review.googlesource.com/1244128
Auto-Submit: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Ben Pastene <bpastene@chromium.org>
Commit-Queue: Ben Pastene <bpastene@chromium.org>

[modify] https://crrev.com/9eb591f4fb296b7217a0ec1ec2e595d95e263ece/scripts/slave/recipe_modules/chromium_tests/chromium_memory.py

Project Member

Comment 23 by bugdroid1@chromium.org, Sep 28

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/33da7cd41e0301772ff68b83bb72840bdddbb002

commit 33da7cd41e0301772ff68b83bb72840bdddbb002
Author: Stephen Martinis <martiniss@chromium.org>
Date: Fri Sep 28 00:11:35 2018

//testing/buildbot: Mixins for webrtc experimental

Use swarming mixins for webrtc experimental bots.

Bug: 880973
Change-Id: I51e386a78f8616a94c9c8a0d67de5bdf096dfe2b
Reviewed-on: https://chromium-review.googlesource.com/1246946
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>
Cr-Commit-Position: refs/heads/master@{#594925}
[modify] https://crrev.com/33da7cd41e0301772ff68b83bb72840bdddbb002/testing/buildbot/chromium.webrtc.fyi.experimental.json
[modify] https://crrev.com/33da7cd41e0301772ff68b83bb72840bdddbb002/testing/buildbot/swarming_mixins.pyl
[modify] https://crrev.com/33da7cd41e0301772ff68b83bb72840bdddbb002/testing/buildbot/waterfalls.pyl

Project Member

Comment 24 by bugdroid1@chromium.org, Sep 28

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/0094aab98c5fba78d3efbe887a23fd93c6f00ce2

commit 0094aab98c5fba78d3efbe887a23fd93c6f00ce2
Author: Stephen Martinis <martiniss@chromium.org>
Date: Fri Sep 28 07:39:17 2018

Remove swarming_dimensions for webrtc experimental bots

These are implemented source side in https://crrev.com/c/1246946.

Bug: 880973
Change-Id: I9602904467d730e70766efc970d9c8a6cd26b6ea
Reviewed-on: https://chromium-review.googlesource.com/1246480
Reviewed-by: Garrett Beaty <gbeaty@chromium.org>
Commit-Queue: Patrik Höglund <phoglund@chromium.org>

[modify] https://crrev.com/0094aab98c5fba78d3efbe887a23fd93c6f00ce2/scripts/slave/recipe_modules/chromium_tests/chromium_webrtc_fyi_experimental.py

Project Member

Comment 25 by bugdroid1@chromium.org, Oct 9

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/clank/internal/apps/+/4de9c50d1ae10da2a7036aaba3bc78d1c71742ed

commit 4de9c50d1ae10da2a7036aaba3bc78d1c71742ed
Author: Stephen Martinis <martiniss@google.com>
Date: Tue Oct 09 18:16:30 2018

Sign in to add a comment