is this bug scoped to switching to task_slices with one task def, or it includes starting using multiple defs?

Include starting using multiple defs, but using task_slices with one def is a prerequisite that would be done as a separate change.

It would be great if no config changes were necessary for most builders. E.g. we already define a named cache for each builder [1], so perhaps that same cache can be specified as a task slice and one would only need to specify how much time most builders should wait for it in the builder_defaults section.

[1]: https://screenshot.googleplex.com/qw6rjFf6qaq

P.S. This will also be super useful for V8 since we've migrated about 40% of CI and 100% of trysever builders to LUCI. As a result this has caused a regression in build runtime due to poor build affinity, e.g. see issue 853313.

Issue 853313 has been merged into this issue.

The way I see it, task slices are not exposed by buildbucket. It should be sufficient to specify named cache expiration/“max wait time”. Then buildbucket can compute task slices based on that. No breaking changes needed

Definitely. I was just saying that it would be nice if one could specify "max wait time" for the automatically-generated cache that swarmbucket provides - see [1] from comment #5.

FWIU defining max wait time for generic named caches, like git, will most probably be a noop because they are universal and present on all bots, even though their contents may be different.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/6b395469313f0cb740cc5b6d2ec7635292d0f065

commit 6b395469313f0cb740cc5b6d2ec7635292d0f065
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Thu Jul 12 20:24:55 2018

[buildbucket] Switch it to use task_slice for Swarming new task request

This is required to use task slice fallback. Implement the fallback in a follow
up to confirm that I got the base working first and reduce the CL complexity.

Bug:  847602 
Change-Id: I7bbceba1cf7a7a32126441f5c3fe713bc6bc6566
Reviewed-on: https://chromium-review.googlesource.com/1110708
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>

[modify] https://crrev.com/6b395469313f0cb740cc5b6d2ec7635292d0f065/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/6b395469313f0cb740cc5b6d2ec7635292d0f065/appengine/cr-buildbucket/swarming/test/swarmbucket_api_test.py
[modify] https://crrev.com/6b395469313f0cb740cc5b6d2ec7635292d0f065/appengine/cr-buildbucket/swarming/test/swarming_test.py

How's this looking?

eeeeee sorry no progress to report

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/8c41ea1c3e4f371588cb47b89aca8bd5ff5f8679

commit 8c41ea1c3e4f371588cb47b89aca8bd5ff5f8679
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Fri Aug 03 19:34:30 2018

buildbucket: add CacheEntry.wait_for_warm_cache_secs

This enables configuring the fallback at the builder level.

Bug:  847602 
Change-Id: Iaa29842b7a7f21740b85eb5254f6ad9b452c95ed
Reviewed-on: https://chromium-review.googlesource.com/1158605
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/8c41ea1c3e4f371588cb47b89aca8bd5ff5f8679/buildbucket/proto/config/project_config.proto

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/5afa6e73b80554bfa3ca9a7c470b255253947ae8

commit 5afa6e73b80554bfa3ca9a7c470b255253947ae8
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Fri Aug 03 20:01:58 2018

Roll infra/go/src/go.chromium.org/luci/ 3f6842cdc..8c41ea1c3 (8 commits)

https://chromium.googlesource.com/infra/luci/luci-go/+log/3f6842cdc472..8c41ea1c3e4f

$ git log 3f6842cdc..8c41ea1c3 --date=short --no-merges --format='%ad %ae %s'
2018-08-03 maruel buildbucket: add CacheEntry.wait_for_warm_cache_secs
2018-08-03 bevc Add initial changes for Crimson UI
2018-08-03 vadimsh [common/sync] Add promise.Map type.
2018-08-02 nodir [buildbucket] Add UpdateBuildRequest.fields
2018-08-02 smut [mpagent] Report metrics about instructions
2018-08-01 jchinlee [buildbucket] Set up annotations->Buildbucket v2 converter.
2018-07-31 bpastene milo: Print (secondary) next to the names of non-primary oncalls.
2018-07-27 nodir [buildbucket] Add Builds RPC mock.

Created with:
  roll-dep infra/go/src/go.chromium.org/luci

TBR=nodir@chromium.org

Bug:  847602 
Change-Id: I523036265b119b64443333538686c94863e3dba5
Reviewed-on: https://chromium-review.googlesource.com/1162647
Reviewed-by: Marc-Antoine Ruel <maruel@chromium.org>
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>

[modify] https://crrev.com/5afa6e73b80554bfa3ca9a7c470b255253947ae8/DEPS

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/33e389d349bf80243a57d910bfce77acb78f2ce0

commit 33e389d349bf80243a57d910bfce77acb78f2ce0
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Tue Aug 07 00:03:40 2018

Marc, can you please provide an update on this? What's done, what's left to do?

It's progressing. :)

Glad to hear that there is progress.

How will it work from users's perspective? What will V8 have to specify in cr-buildbucket.cfg to make cache affinity work? Or will it just work (c) for the default builder cache case?

Also I am somewhat surprised that there is just one task slice defined in https://chrome-internal.googlesource.com/infradata/config/+/master/configs/cr-buildbucket-dev/swarming_task_template_canary.json. Shouldn't it have another task slice without builder_cache?

That's the template. Buildbucket decides at evaluation how many fallbacks to create.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/a5ea19215c37a35ab1d502d81eab1b64fd1784a0

commit a5ea19215c37a35ab1d502d81eab1b64fd1784a0
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Wed Aug 08 16:38:07 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/fb9847fb95c68c6f3fb4d80d0e81e1ce69b5bc2c

commit fb9847fb95c68c6f3fb4d80d0e81e1ce69b5bc2c
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Wed Aug 08 17:44:24 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/672a3976d0e6a8fce0e38262fa0f20c870681fb7

commit 672a3976d0e6a8fce0e38262fa0f20c870681fb7
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Wed Aug 08 19:37:48 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/0e27289921204741e1d653f8dc545ab1298d346f

commit 0e27289921204741e1d653f8dc545ab1298d346f
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Wed Aug 08 22:47:00 2018

[cr-buildbucket] stop supporting old swarming task template format

All task templates have been updated.

R=nodir@chromium.org

Bug:  847602 
Change-Id: I31a7347ad9457fe7dc1276dcc7964a9062b18f4d
Reviewed-on: https://chromium-review.googlesource.com/1168082
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/0e27289921204741e1d653f8dc545ab1298d346f/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/0e27289921204741e1d653f8dc545ab1298d346f/appengine/cr-buildbucket/swarming/test/swarming_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/abdba21cbf4c2f16d88e6b6b9248008fc9f1ed63

commit abdba21cbf4c2f16d88e6b6b9248008fc9f1ed63
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Fri Aug 10 22:04:58 2018

[cr-buildbucket] named cache fallback via CacheEntry.wait_for_warm_cache_secs

Update protos from go.chromium.org/luci that was just rolled in in previous
commit.

Bug:  847602 
Change-Id: I5c48841a234864ee5d8d5064dede44d6c4f976b4
Reviewed-on: https://chromium-review.googlesource.com/1158606
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>

[modify] https://crrev.com/abdba21cbf4c2f16d88e6b6b9248008fc9f1ed63/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/abdba21cbf4c2f16d88e6b6b9248008fc9f1ed63/appengine/cr-buildbucket/proto/config/project_config_pb2.py
[modify] https://crrev.com/abdba21cbf4c2f16d88e6b6b9248008fc9f1ed63/appengine/cr-buildbucket/swarming/test/swarming_test.py
[modify] https://crrev.com/abdba21cbf4c2f16d88e6b6b9248008fc9f1ed63/appengine/cr-buildbucket/swarming/test/swarmingcfg_test.py
[modify] https://crrev.com/abdba21cbf4c2f16d88e6b6b9248008fc9f1ed63/appengine/cr-buildbucket/swarming/swarmingcfg.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/5ebd60561191547ff686b75ff33914045a48dcec

commit 5ebd60561191547ff686b75ff33914045a48dcec
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Wed Aug 15 19:23:05 2018

[cr-buildbucket] named cache fallback in Swarming task template

This will simplify our life for the immediate needs by making a one-size
fits all named cache fallback on the builder cache, which then can be
ovveridden by builder specific configuration.

Bug:  847602 
Change-Id: Id662d705e7995bac07e0099299ae0d895bc5cf85
Reviewed-on: https://chromium-review.googlesource.com/1175421
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>

[modify] https://crrev.com/5ebd60561191547ff686b75ff33914045a48dcec/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/5ebd60561191547ff686b75ff33914045a48dcec/appengine/cr-buildbucket/swarming/test/swarming_test.py

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/a1af925b7e59b061266a1b19af5fdb829c73b72d

commit a1af925b7e59b061266a1b19af5fdb829c73b72d
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Thu Aug 16 14:18:07 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/6294c06467d2348206758998e5b3e21f1ddccb6e

commit 6294c06467d2348206758998e5b3e21f1ddccb6e
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Thu Aug 16 15:14:04 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/a28bc07ab3aa1f38bee5d7d282069b0992863da2

commit a28bc07ab3aa1f38bee5d7d282069b0992863da2
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Thu Aug 16 15:48:19 2018

[cr-buildbucket] fix typo in named cache fallback dimension name

The bots are exposing the named caches as dimension 'caches', not
'cache'.

It didn't break anything, as the Swarming server used immediate fallback
(aka NO_RESOURCE) and fell to the cold cache. This change fixes this and
will make the warm cache work.

It's a fairly safe change, since it's changing the injected dimension
key only for the hot task slice.

R=nodir@chromium.org

Bug:  847602 
Change-Id: Iac7a2914379d31cabd7a12b7d4841188f633c5e9
Reviewed-on: https://chromium-review.googlesource.com/1177903
Reviewed-by: Nodir Turakulov <nodir@chromium.org>
Commit-Queue: Marc-Antoine Ruel <maruel@chromium.org>

[modify] https://crrev.com/a28bc07ab3aa1f38bee5d7d282069b0992863da2/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/a28bc07ab3aa1f38bee5d7d282069b0992863da2/appengine/cr-buildbucket/swarming/test/swarming_test.py

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/df0da2424e461beadf9e093a643b1799984a43ba

commit df0da2424e461beadf9e093a643b1799984a43ba
Author: Marc-Antoine Ruel <maruel@chromium.org>
Date: Thu Aug 16 16:43:10 2018

The named cache fallback is done. I'm following bot affinity in  issue 880550 .