the plan is to define pools of machines where we will schedule tasks that require "git_infra" cache to be present. This is a manual process and I don't know how many machines to put to those pools. I could add 3 of each platform, similar to what we had on buildbot, but I think it will increase current wait time from the user perspective, because currently tasks starts fast but runs slowly (because there is no cache). I cannot tune the number of machines because we don't have a graph that would display current wait time per builder. We need to build that graph first.

to simplify this we can start with generic git cache named "git_cache" that would contain any repositories, not specifically infra. In that case it is OK to define a pool yet, but schedule tasks to any machines, because a generic git_cache will be reused by different projects. WDYT?

In practice only infra trybots run recipes on Swarming. So if we add a new git cache, without restricting pool of machines, this git cache directory will randomly show up on various Swarming bots, never to be used again (the probability of same bot to be frequently used by infra tryjobs is relatively low, there are many bots).

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config.git/+/a1f8e176641bae44a29a36d7a1f6219fac3f262a

commit a1f8e176641bae44a29a36d7a1f6219fac3f262a
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Nov 04 21:33:28 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config.git/+/fa85c3beaa89410f6e151dcb08aa28d44a78c5eb

commit fa85c3beaa89410f6e151dcb08aa28d44a78c5eb
Author: Nodir Turakulov <nodir@google.com>
Date: Fri Nov 11 23:48:04 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/f69b6d18d6aea88c785697672900f75fc7dd1d29

commit f69b6d18d6aea88c785697672900f75fc7dd1d29
Author: Nodir Turakulov <nodir@chromium.org>
Date: Fri Nov 11 23:43:30 2016

kitchen: pass cache_dir property

Add -cache-dir flag. If passed, make it OS-native, absolute and pass as
cache_dir property.

Swarmbucket will specify "rundir/caches" which will contain named caches
for git, isolate and goma

R=maruel@chromium.org, phosek@chromium.org
BUG=660938,  661390 

Change-Id: I8e80155dc2f2283c2beda52b5009d4130d762693
Reviewed-on: https://chromium-review.googlesource.com/410069
Reviewed-by: Erik Staab <estaab@chromium.org>
Commit-Queue: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/f69b6d18d6aea88c785697672900f75fc7dd1d29/go/src/infra/tools/kitchen/cook.go

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config.git/+/87ff32e9e70c12726b8fc0d524228ba8d4995b1c

commit 87ff32e9e70c12726b8fc0d524228ba8d4995b1c
Author: Nodir Turakulov <nodir@google.com>
Date: Sat Nov 12 04:25:59 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config.git/+/4769d1a474924789b51fd206160348b0493fff1d

commit 4769d1a474924789b51fd206160348b0493fff1d
Author: Nodir Turakulov <nodir@google.com>
Date: Thu Nov 17 21:17:07 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config.git/+/98794146900e2ccc4e590a73db44aef7e79c2e14

commit 98794146900e2ccc4e590a73db44aef7e79c2e14
Author: Nodir Turakulov <nodir@google.com>
Date: Mon Nov 21 09:02:25 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config.git/+/9eb66bcdb6024449a5680a953c9cd7b4d1deee80

commit 9eb66bcdb6024449a5680a953c9cd7b4d1deee80
Author: Nodir Turakulov <nodir@google.com>
Date: Mon Nov 21 09:38:07 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config.git/+/99f6a361c7d92ec974befafc0563506a8afb5db1

commit 99f6a361c7d92ec974befafc0563506a8afb5db1
Author: Nodir Turakulov <nodir@google.com>
Date: Mon Nov 21 09:31:35 2016

This still does not work on Windows. Investigating.

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/a93ce5abfcc7054c20de9cbaf296e1176bd0155e

commit a93ce5abfcc7054c20de9cbaf296e1176bd0155e
Author: nodir <nodir@chromium.org>
Date: Wed Feb 01 18:10:19 2017

bot_code: Add c dir to PASSLIST

PASSLIST in bot_main.py defines what may stay in the swarming bot root.
The named caches root wasn't in the PASSLIST, so it was deleted on each
bot start.

Add it there.

R=maruel@chromium.org, vadimsh@chromium.org
BUG= 661390 

Review-Url: https://codereview.chromium.org/2669833002

[modify] https://crrev.com/a93ce5abfcc7054c20de9cbaf296e1176bd0155e/appengine/swarming/swarming_bot/bot_code/bot_main.py

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/a93ce5abfcc7054c20de9cbaf296e1176bd0155e

commit a93ce5abfcc7054c20de9cbaf296e1176bd0155e
Author: nodir <nodir@chromium.org>
Date: Wed Feb 01 18:10:19 2017

bot_code: Add c dir to PASSLIST

PASSLIST in bot_main.py defines what may stay in the swarming bot root.
The named caches root wasn't in the PASSLIST, so it was deleted on each
bot start.

Add it there.

R=maruel@chromium.org, vadimsh@chromium.org
BUG= 661390 

Review-Url: https://codereview.chromium.org/2669833002

[modify] https://crrev.com/a93ce5abfcc7054c20de9cbaf296e1176bd0155e/appengine/swarming/swarming_bot/bot_code/bot_main.py

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/fb391a2976db77b810e1178c0aaed972d6eaf4f4

commit fb391a2976db77b810e1178c0aaed972d6eaf4f4
Author: nodir <nodir@chromium.org>
Date: Thu Feb 02 03:22:37 2017

implement unlink on Windows

unlink on Windows uses DeleteFileW and RemoveDirectoryW to delete
the symlink and not the link source.

Relevant MSDN articles:
  CreateSymbolicLink: https://msdn.microsoft.com/en-us/library/windows/desktop/aa363866(v=vs.85).aspx
  DeleteFile: https://msdn.microsoft.com/en-us/library/windows/desktop/aa363915(v=vs.85).aspx
  RemoveDirectory: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365488(v=vs.85).aspx

R=maruel@chromium.org, vadimsh@chromium.org
BUG= 661390 

Review-Url: https://codereview.chromium.org/2668503008

[modify] https://crrev.com/fb391a2976db77b810e1178c0aaed972d6eaf4f4/client/utils/fs.py

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/fb391a2976db77b810e1178c0aaed972d6eaf4f4

commit fb391a2976db77b810e1178c0aaed972d6eaf4f4
Author: nodir <nodir@chromium.org>
Date: Thu Feb 02 03:22:37 2017

implement unlink on Windows

unlink on Windows uses DeleteFileW and RemoveDirectoryW to delete
the symlink and not the link source.

Relevant MSDN articles:
  CreateSymbolicLink: https://msdn.microsoft.com/en-us/library/windows/desktop/aa363866(v=vs.85).aspx
  DeleteFile: https://msdn.microsoft.com/en-us/library/windows/desktop/aa363915(v=vs.85).aspx
  RemoveDirectory: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365488(v=vs.85).aspx

R=maruel@chromium.org, vadimsh@chromium.org
BUG= 661390 

Review-Url: https://codereview.chromium.org/2668503008

[modify] https://crrev.com/fb391a2976db77b810e1178c0aaed972d6eaf4f4/client/utils/fs.py

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/fb391a2976db77b810e1178c0aaed972d6eaf4f4

commit fb391a2976db77b810e1178c0aaed972d6eaf4f4
Author: nodir <nodir@chromium.org>
Date: Thu Feb 02 03:22:37 2017

implement unlink on Windows

unlink on Windows uses DeleteFileW and RemoveDirectoryW to delete
the symlink and not the link source.

Relevant MSDN articles:
  CreateSymbolicLink: https://msdn.microsoft.com/en-us/library/windows/desktop/aa363866(v=vs.85).aspx
  DeleteFile: https://msdn.microsoft.com/en-us/library/windows/desktop/aa363915(v=vs.85).aspx
  RemoveDirectory: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365488(v=vs.85).aspx

R=maruel@chromium.org, vadimsh@chromium.org
BUG= 661390 

Review-Url: https://codereview.chromium.org/2668503008

[modify] https://crrev.com/fb391a2976db77b810e1178c0aaed972d6eaf4f4/client/utils/fs.py

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/a6efc1e4f6920dd1249ffb2bff3bf66ec6631e7d

commit a6efc1e4f6920dd1249ffb2bff3bf66ec6631e7d
Author: nodir <nodir@chromium.org>
Date: Thu Feb 02 21:03:35 2017

swarming: delete named cache symlinks before workdir

On Windows, shutil.rmtree is used to delete the workdir which removes
the actual named cache directories.

Unlink the symlinks before deleting the workdir

R=maruel@chromium.org, vadimsh@chromium.org
BUG= 661390 

Review-Url: https://codereview.chromium.org/2663023004

[modify] https://crrev.com/a6efc1e4f6920dd1249ffb2bff3bf66ec6631e7d/client/named_cache.py
[modify] https://crrev.com/a6efc1e4f6920dd1249ffb2bff3bf66ec6631e7d/client/run_isolated.py
[modify] https://crrev.com/a6efc1e4f6920dd1249ffb2bff3bf66ec6631e7d/client/tests/run_isolated_test.py

git cache is reused on windows machines https://luci-milo.appspot.com/swarming/task/341a7cc3aad01610

https://s-media-cache-ak0.pinimg.com/originals/fa/a1/b6/faa1b62f816f346a673d731f1007773e.gif