New issue
Advanced search Search tips

Issue 823434 link

Starred by 3 users
Status: Started
Owner:
iannu...@google.com
Cc:
mar...@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocking:
issue 823399
issue 911511



Sign in to add a comment

Implement Swarming Pool Task Template feature

Project Member Reported by iannucci@chromium.org, Mar 19 2018 
Design doc: https://goo.gl/UaBCaA

Design summary:

Add a feature to swarming's pool.cfg configuration to allow the association of a 'task template' (specifying e.g. CIPD packages, envvars, etc.) with a swarming pool. New tasks created in the pool will adopt the task template as a base, and then be merged on top of the template, before calculating the task hash and scheduling it.
Project Member

Comment 2 by bugdroid1@chromium.org, Apr 7 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/52f0a9484ef2255e2b964799bced18faaf738d1e

commit 52f0a9484ef2255e2b964799bced18faaf738d1e
Author: Robert Iannucci <iannucci@chromium.org>
Date: Sat Apr 07 00:19:10 2018

[swarming] Add evaluate_only property to NewTaskRequest.

This will become more useful when the task template feature is finished,
but it's already semi-useful to see the effects of the server-wide
properties (e.g. cipd version and such).

R=maruel@chromium.org, nodir@chromium.org, vadimsh@chromium.org

Bug: 823434
Change-Id: I48d5be61bfe4778f5601680605776a9e10770f89
Reviewed-on: https://chromium-review.googlesource.com/985647
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>

[modify] https://crrev.com/52f0a9484ef2255e2b964799bced18faaf738d1e/appengine/swarming/handlers_endpoints.py
[modify] https://crrev.com/52f0a9484ef2255e2b964799bced18faaf738d1e/appengine/swarming/handlers_endpoints_test.py
[modify] https://crrev.com/52f0a9484ef2255e2b964799bced18faaf738d1e/appengine/swarming/swarming_rpcs.py
Project Member

Comment 3 by bugdroid1@chromium.org, Apr 12 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/62fda9e9c2f948f656c30dfa9c7b13475025fa9d

commit 62fda9e9c2f948f656c30dfa9c7b13475025fa9d
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Thu Apr 12 16:08:37 2018

[swarming] Run TaskRequest._pre_put_hook() for eval-only

Many validity checks for a TaskRequest is done in TaskRequest._pre_put_hook().
When evaluate_only is used, these checks were not run because the entity is not
stored. Call the function manually.

TBR=iannucci@chromium.org
Bug: 823434
Change-Id: I915345452a2791a39133d033a411b6ada9dafad5
Reviewed-on: https://chromium-review.googlesource.com/1009004
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>

[modify] https://crrev.com/62fda9e9c2f948f656c30dfa9c7b13475025fa9d/appengine/swarming/handlers_endpoints.py
Project Member

Comment 4 by bugdroid1@chromium.org, Apr 16 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/b56c77e75ab33d686cb59ed3aed0de1572fac630

commit b56c77e75ab33d686cb59ed3aed0de1572fac630
Author: Robert Iannucci <iannucci@chromium.org>
Date: Mon Apr 16 18:17:49 2018

[swarming] Add configuration (including validation) for task templates.

This updates pools.cfg to accept task template configuration, but
doesn't do anything with it, other than parse+verify the config.

R=maruel@chromium.org, nodir@chromium.org, vadimsh@chromium.org

Bug: 823434
Change-Id: Ib5833d090a8b9d0f8536b5b9bfc7e513ec3bdf40
Reviewed-on: https://chromium-review.googlesource.com/982375
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/b56c77e75ab33d686cb59ed3aed0de1572fac630/appengine/swarming/proto/pools.proto
[modify] https://crrev.com/b56c77e75ab33d686cb59ed3aed0de1572fac630/appengine/swarming/proto/pools_pb2.py
[add] https://crrev.com/b56c77e75ab33d686cb59ed3aed0de1572fac630/appengine/swarming/server/directory_occlusion.py
[add] https://crrev.com/b56c77e75ab33d686cb59ed3aed0de1572fac630/appengine/swarming/server/directory_occlusion_test.py
[modify] https://crrev.com/b56c77e75ab33d686cb59ed3aed0de1572fac630/appengine/swarming/server/pools_config.py
[modify] https://crrev.com/b56c77e75ab33d686cb59ed3aed0de1572fac630/appengine/swarming/server/pools_config_test.py
[modify] https://crrev.com/b56c77e75ab33d686cb59ed3aed0de1572fac630/appengine/swarming/server/task_scheduler_test.py
[modify] https://crrev.com/b56c77e75ab33d686cb59ed3aed0de1572fac630/appengine/swarming/test_env_handlers.py
Project Member

Comment 5 by bugdroid1@chromium.org, Jun 13 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-py.git/+/579883b934bd81e66e69fca15bfd6809884badf9

commit 579883b934bd81e66e69fca15bfd6809884badf9
Author: Robert Iannucci <iannucci@chromium.org>
Date: Wed Jun 13 17:08:17 2018

[swarming] Actually apply pool based TaskTemplates.

This extends the TaskRequest with a new 'pool_task_template' enum,
allowing the caller to control the application of the pool's prod/canary
TaskTemplate to the task.

This also implements the actual TaskTemplate->TaskProperties merging
logic.

R=maruel@chromium.org, nodir@chromium.org

Bug: 823434
Change-Id: I54dd287be4371c8def6099a2dce9f763a8572977
Reviewed-on: https://chromium-review.googlesource.com/998798
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>

[modify] https://crrev.com/579883b934bd81e66e69fca15bfd6809884badf9/appengine/swarming/handlers_endpoints.py
[modify] https://crrev.com/579883b934bd81e66e69fca15bfd6809884badf9/appengine/swarming/handlers_endpoints_test.py
[modify] https://crrev.com/579883b934bd81e66e69fca15bfd6809884badf9/appengine/swarming/handlers_test.py
[modify] https://crrev.com/579883b934bd81e66e69fca15bfd6809884badf9/appengine/swarming/message_conversion.py
[modify] https://crrev.com/579883b934bd81e66e69fca15bfd6809884badf9/appengine/swarming/server/pools_config.py
[modify] https://crrev.com/579883b934bd81e66e69fca15bfd6809884badf9/appengine/swarming/server/pools_config_test.py
[modify] https://crrev.com/579883b934bd81e66e69fca15bfd6809884badf9/appengine/swarming/server/service_accounts_test.py
[modify] https://crrev.com/579883b934bd81e66e69fca15bfd6809884badf9/appengine/swarming/server/task_queues_test.py
[modify] https://crrev.com/579883b934bd81e66e69fca15bfd6809884badf9/appengine/swarming/server/task_request.py
[modify] https://crrev.com/579883b934bd81e66e69fca15bfd6809884badf9/appengine/swarming/server/task_request_test.py
[modify] https://crrev.com/579883b934bd81e66e69fca15bfd6809884badf9/appengine/swarming/server/task_result_test.py
[modify] https://crrev.com/579883b934bd81e66e69fca15bfd6809884badf9/appengine/swarming/server/task_scheduler.py
[modify] https://crrev.com/579883b934bd81e66e69fca15bfd6809884badf9/appengine/swarming/server/task_scheduler_test.py
[modify] https://crrev.com/579883b934bd81e66e69fca15bfd6809884badf9/appengine/swarming/server/task_to_run_test.py
[modify] https://crrev.com/579883b934bd81e66e69fca15bfd6809884badf9/appengine/swarming/swarming_rpcs.py
[modify] https://crrev.com/579883b934bd81e66e69fca15bfd6809884badf9/appengine/swarming/test_env_handlers.py
[modify] https://crrev.com/579883b934bd81e66e69fca15bfd6809884badf9/client/swarming.py
[modify] https://crrev.com/579883b934bd81e66e69fca15bfd6809884badf9/client/tests/swarming_test.py
Project Member

Comment 6 by bugdroid1@chromium.org, Jun 13 2018 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/37a7d771e21f46c53ef237d2f7a1dc465dd59de4

commit 37a7d771e21f46c53ef237d2f7a1dc465dd59de4
Author: Robert Iannucci <iannucci@chromium.org>
Date: Wed Jun 13 21:44:55 2018

Comment 7 by iannucci@chromium.org, Jun 15 2018

Blocking: 823399
Project Member

Comment 8 by bugdroid1@chromium.org, Jun 15 2018 

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/2a8c11ce5d733cead8b6ce80c31bfed950c5e864

commit 2a8c11ce5d733cead8b6ce80c31bfed950c5e864
Author: Robert Iannucci <iannucci@chromium.org>
Date: Fri Jun 15 23:56:46 2018
Project Member

Comment 10 by bugdroid1@chromium.org, Jun 29 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/depot_tools/+/a19649b86b3d0f6ab434ba1a8f75e0271cf117d8

commit a19649b86b3d0f6ab434ba1a8f75e0271cf117d8
Author: Robert Iannucci <iannucci@chromium.org>
Date: Fri Jun 29 16:31:45 2018

Clean up cache_dir handling in gclient/git_cache/bot_update.

This CL makes a couple changes:
  * The goal here is to be able to specify git cache entirely from the
    environment variable $GIT_CACHE_PATH and not require special treatment from
    bot_update. Eventually this will be specified at the swarming task level
    instead of in the recipe (i.e. "cached git" will eventually be an
    implementation detail of git on the bots and completely transparent to
    all other software).
  * Removal of the general --cache-dir option from gclient. This option was
    error-prone; it doesn't actually make sense to configure this on
    a per-invocation basis. The sole exception was `gclient config`, which
    now allows this option to be set directly.
  * Consolidation of GitWrapper.cache_dir and GitWrapper._GetMirror; previously
    these two things could disagree, leading to weird intermediate states. Now
    they're the same value.

R=agable@chromium.org, ehmaldonado@chromium.org, tandrii@chromium.org

Bug: 823434
Change-Id: I14adc7619b5fc10768ce32be2651c6215ba94aff
Reviewed-on: https://chromium-review.googlesource.com/1105473
Reviewed-by: Aaron Gable <agable@chromium.org>
Reviewed-by: Edward Lesmes <ehmaldonado@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>

[modify] https://crrev.com/a19649b86b3d0f6ab434ba1a8f75e0271cf117d8/gclient.py
[modify] https://crrev.com/a19649b86b3d0f6ab434ba1a8f75e0271cf117d8/tests/git_cache_test.py
[modify] https://crrev.com/a19649b86b3d0f6ab434ba1a8f75e0271cf117d8/gclient_scm.py
[modify] https://crrev.com/a19649b86b3d0f6ab434ba1a8f75e0271cf117d8/git_cache.py
[modify] https://crrev.com/a19649b86b3d0f6ab434ba1a8f75e0271cf117d8/tests/gclient_smoketest.py
Project Member

Comment 11 by bugdroid1@chromium.org, Jun 29 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1a4c4c2b5ed9eedb38f2441f880d6e01120e49b1

commit 1a4c4c2b5ed9eedb38f2441f880d6e01120e49b1
Author: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Date: Fri Jun 29 18:18:29 2018

Roll src/third_party/depot_tools 406de133efaf..a19649b86b3d (1 commits)

https://chromium.googlesource.com/chromium/tools/depot_tools.git/+log/406de133efaf..a19649b86b3d


git log 406de133efaf..a19649b86b3d --date=short --no-merges --format='%ad %ae %s'
2018-06-29 iannucci@chromium.org Clean up cache_dir handling in gclient/git_cache/bot_update.


Created with:
  gclient setdep -r src/third_party/depot_tools@a19649b86b3d

The AutoRoll server is located here: https://depot-tools-chromium-roll.skia.org

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.



BUG=chromium:823434
TBR=agable@chromium.org

Change-Id: I3003d9a0a791da948c861cb0156cf924fb6864e9
Reviewed-on: https://chromium-review.googlesource.com/1120411
Reviewed-by: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Commit-Queue: depot-tools-chromium-autoroll <depot-tools-chromium-autoroll@skia-buildbots.google.com.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#571551}
[modify] https://crrev.com/1a4c4c2b5ed9eedb38f2441f880d6e01120e49b1/DEPS
Project Member

Comment 12 by bugdroid1@chromium.org, Jul 4 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/1ab9df1ef9532c1669a4658215799f132ce429ec

commit 1ab9df1ef9532c1669a4658215799f132ce429ec
Author: Robert Iannucci <iannucci@chromium.org>
Date: Wed Jul 04 02:31:18 2018

Start implementing cross-platform support in third_party_packages.

This will be used to build versions of cpython and git for ARM & MIPS
linux platforms so that the Chrome Operations Foundation software layer
can be uniformly distributed on swarming.

R=tandrii@chromium.org, vadimsh@chromium.org

Bug: 823434
Change-Id: I7f16d32426e30a585642946fc9dc89359d3d4674
Reviewed-on: https://chromium-review.googlesource.com/1125199
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[add] https://crrev.com/1ab9df1ef9532c1669a4658215799f132ce429ec/recipes/recipes/third_party_packages.expected/cross_compile.json
[modify] https://crrev.com/1ab9df1ef9532c1669a4658215799f132ce429ec/recipes/recipes/third_party_packages.py
[modify] https://crrev.com/1ab9df1ef9532c1669a4658215799f132ce429ec/recipes/README.recipes.md
[modify] https://crrev.com/1ab9df1ef9532c1669a4658215799f132ce429ec/recipes/recipe_modules/third_party_packages/examples/python.py
[modify] https://crrev.com/1ab9df1ef9532c1669a4658215799f132ce429ec/recipes/recipe_modules/third_party_packages/api.py
[add] https://crrev.com/1ab9df1ef9532c1669a4658215799f132ce429ec/recipes/recipe_modules/third_party_packages/examples/python.expected/new_on_linux-amd64-for-linux-arm64.json
Project Member

Comment 13 by bugdroid1@chromium.org, Jul 8 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/ecdc034ac519e600001c536ebb1fb779dcbef08f

commit ecdc034ac519e600001c536ebb1fb779dcbef08f
Author: Robert Iannucci <iannucci@chromium.org>
Date: Sun Jul 08 00:27:41 2018

Strip dockerbuild of all external python dependencies.

Also include it in the recipe bundle.

R=tandrii@chromium.org, vadimsh@chromium.org

Bug: 823434
Change-Id: I6e31531b37d1f62f6ca60a611d03386adc7e8363
Reviewed-on: https://chromium-review.googlesource.com/1125519
Commit-Queue: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/ecdc034ac519e600001c536ebb1fb779dcbef08f/.gitattributes
[modify] https://crrev.com/ecdc034ac519e600001c536ebb1fb779dcbef08f/infra/tools/dockerbuild/__main__.py

Comment 15 by iannu...@google.com, Oct 18

Owner: iannu...@google.com

Comment 16 by iannu...@google.com, Dec 6

Blocking: 911511
►

Sign in to add a comment