I'm not sure if there's anything we can do about this, but filing in case there is...

===

Basically the issue is that cherry-picking changes to the kernel from gerrit is slow.  

For instance, if I got to:

https://chromium-review.googlesource.com/c/508834/

...and I go to "Download" and get the cherry-pick text, I see:

  git fetch https://chromium.googlesource.com/chromiumos/third_party/kernel refs/changes/34/508834/1 && git cherry-pick FETCH_HEAD

---

Running this takes 18 seconds:

$ time git fetch https://chromium.googlesource.com/chromiumos/third_party/kernel refs/changes/34/508834/1
remote: Counting objects: 58060, done
remote: Finding sources: 100% (4/4)
remote: Total 4 (delta 0), reused 2 (delta 0)
Unpacking objects: 100% (4/4), done.
From https://chromium.googlesource.com/chromiumos/third_party/kernel
 * branch                      refs/changes/34/508834/1 -> FETCH_HEAD

real    0m18.625s
user    0m5.572s
sys     0m6.455s

---

OTOH, running this is much faster:

$ time curl https://chromium-review.googlesource.com/changes/508834/revisions/1/patch?download > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2792    0  2792    0     0  15931      0 --:--:-- --:--:-- --:--:-- 15954

real    0m1.829s
user    0m0.012s
sys     0m0.006s

---

It appears that fetching the same change a 2nd time is a bit faster, but still not quite as fast.  Re-running the two commands above:

$ time git fetch https://chromium.googlesource.com/chromiumos/third_party/kernel refs/changes/34/508834/1
From https://chromium.googlesource.com/chromiumos/third_party/kernel
 * branch                      refs/changes/34/508834/1 -> FETCH_HEAD

real    0m3.982s
user    0m1.249s
sys     0m0.324s

$ time curl https://chromium-review.googlesource.com/changes/508834/revisions/1/patch?download > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2792    0  2792    0     0  10801      0 --:--:-- --:--:-- --:--:-- 10821

real    0m0.273s
user    0m0.011s
sys     0m0.011s

===

This speed (plus the stupid way that our builder cherry-picks changes) is likely the root cause of the slow Commit Queue Sync of guado moblab build 5814.  A small snippet from there:

[patch_branch 0b095aa7129c] UPSTREAM: Bluetooth: Introduce helper functions for socket cookie handling
 Author: Marcel Holtmann <marcel@holtmann.org>
 Date: Tue Aug 30 05:00:34 2016 +0200
 1 file changed, 29 insertions(+), 12 deletions(-)
13:56:54: INFO: RunCommand: git fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel refs/changes/14/475414/7 in /b/cbuild/repository/src/third_party/kernel/v3.14
13:57:14: INFO: Attempting to cherry-pick change chromiumos/third_party/kernel:refs/changes/14/475414/7:ca8d1056 "UPSTREAM: Bluetooth: Use numbers for subsystem version string"
Trying simple merge.

[patch_branch f878a33be370] UPSTREAM: Bluetooth: Use numbers for subsystem version string
 Author: Marcel Holtmann <marcel@holtmann.org>
 Date: Tue Aug 30 05:00:35 2016 +0200
 3 files changed, 11 insertions(+), 6 deletions(-)
13:57:16: INFO: RunCommand: git fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel refs/changes/15/475415/7 in /b/cbuild/repository/src/third_party/kernel/v3.14
13:57:36: INFO: Attempting to cherry-pick change chromiumos/third_party/kernel:refs/changes/15/475415/7:d71f71b2 "UPSTREAM: Bluetooth: Send control open and close only when cookie is present"
Trying simple merge.

[patch_branch 9e8267473a35] UPSTREAM: Bluetooth: Send control open and close only when cookie is present
 Author: Marcel Holtmann <marcel@holtmann.org>
 Date: Tue Aug 30 05:00:36 2016 +0200
 1 file changed, 16 insertions(+), 2 deletions(-)
13:57:39: INFO: RunCommand: git fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel refs/changes/16/475416/7 in /b/cbuild/repository/src/third_party/kernel/v3.14
13:58:00: INFO: Attempting to cherry-pick change chromiumos/third_party/kernel:refs/changes/16/475416/7:b3c05b98 "UPSTREAM: Bluetooth: Assign the channel early when binding HCI sockets"
Trying simple merge.

[patch_branch ba8845c2e927] UPSTREAM: Bluetooth: Assign the channel early when binding HCI sockets
 Author: Marcel Holtmann <marcel@holtmann.org>
 Date: Tue Aug 30 05:00:37 2016 +0200
 1 file changed, 11 insertions(+), 5 deletions(-)
13:58:02: INFO: RunCommand: git fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel refs/changes/17/475417/7 in /b/cbuild/repository/src/third_party/kernel/v3.14
13:58:24: INFO: Attempting to cherry-pick change chromiumos/third_party/kernel:refs/changes/17/475417/7:123eeba3 "UPSTREAM: Bluetooth: Add extra channel checks for control open/close messages"
Trying simple merge.

[patch_branch b9140edf5125] UPSTREAM: Bluetooth: Add extra channel checks for control open/close messages
 Author: Marcel Holtmann <marcel@holtmann.org>
 Date: Tue Aug 30 05:00:38 2016 +0200
 1 file changed, 19 insertions(+), 5 deletions(-)
13:58:26: INFO: RunCommand: git fetch -f https://chromium-review.googlesource.com/chromiumos/third_party/kernel refs/changes/18/475418/7 in /b/cbuild/repository/src/third_party/kernel/v3.14
13:58:46: INFO: Attempting to cherry-pick change chromiumos/third_party/kernel:refs/changes/18/475418/7:44a2123c "UPSTREAM: Bluetooth: Send control open and close messages for HCI raw sockets"
Trying simple merge.


That run has 106 kernel changes to pick (plus some other changes).  At ~23 seconds a change it means we spent ~39 minutes patching in kernel CLs.  Checking against reality:

13:54:18 - start of patching in changes (not just kernel changes).
14:44:54 - end of patching in changes (not just kernel changes).

...so that's about 50 minutes.

===

I'll file a separate bug about making our build bot smarter (there are some tricks we can do) and put a reference here...