Design proposal and discussion: https://docs.google.com/document/d/1YVbRMg60PSdzXos9VZiOB0-JzM0_v9u8MvZSpFD0TKs

Detailed design: goto/findit-buildaheads-design

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/2219d091a3c16edd6b4ca0a2c6e35867630eb54d

commit 2219d091a3c16edd6b4ca0a2c6e35867630eb54d
Author: Roberto Carrillo <robertocn@google.com>
Date: Wed Feb 21 01:03:41 2018

[Findit] Index for getting recent tryjobs for named cache.

With this index, we'll be able to query for a given cache name (which is
derived from the name of the builder in the waterfall) which are the
most recent jobs completed.

This will be needed to prioritize build-aheads.

R=chanli,stgao,lijeffrey,wylieb
BUG=810959

Change-Id: I4be860a1eb27c5c6150b7e1ce89de37e686295b9
Reviewed-on: https://chromium-review.googlesource.com/926786
Reviewed-by: Shuotao Gao <stgao@chromium.org>
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>

[modify] https://crrev.com/2219d091a3c16edd6b4ca0a2c6e35867630eb54d/appengine/findit/index.yaml

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/0b8e38aef3b2782d58ac6f3c8338dd6e634356a6

commit 0b8e38aef3b2782d58ac6f3c8338dd6e634356a6
Author: Roberto Carrillo <robertocn@google.com>
Date: Fri Feb 23 22:56:01 2018

[Findit] Implement GetCommitsInLastHour.

This function will be used by the upcoming cron job to do full builds on
the bots' caches before they are needed. It will be used to determine if
it is a good time to schedule the builds, as it is thought that the
least bad time to use these bots for such tasks is when nobody is
committing code to the repo.

R=chanli,stgao,lijeffrey,wylieb

Bug: 810959
Change-Id: Idddc686c7c5400c428b2c4e4d01aa888efc9d8c2
Reviewed-on: https://chromium-review.googlesource.com/917461
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>
Reviewed-by: Shuotao Gao <stgao@chromium.org>
Reviewed-by: Chan Li <chanli@chromium.org>

[modify] https://crrev.com/0b8e38aef3b2782d58ac6f3c8338dd6e634356a6/appengine/findit/libs/gitiles/test/gitiles_repository_test.py
[modify] https://crrev.com/0b8e38aef3b2782d58ac6f3c8338dd6e634356a6/appengine/findit/services/test/git_test.py
[modify] https://crrev.com/0b8e38aef3b2782d58ac6f3c8338dd6e634356a6/appengine/findit/libs/gitiles/git_repository.py
[modify] https://crrev.com/0b8e38aef3b2782d58ac6f3c8338dd6e634356a6/appengine/findit/libs/gitiles/gitiles_repository.py
[modify] https://crrev.com/0b8e38aef3b2782d58ac6f3c8338dd6e634356a6/appengine/findit/services/git.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/00f8e25084e3cf7b47163b5fa5fb1ecfc5c5893d

commit 00f8e25084e3cf7b47163b5fa5fb1ecfc5c5893d
Author: Roberto Carrillo <robertocn@google.com>
Date: Tue Feb 27 23:09:09 2018

[Findit] Add function to get supported compile builders by platform.

This is needed by the build-aheads to figure out which caches to
pre-build. This current naive approach assumes that builders are compile
builders if they have "[B|b]uild" in their names, or are under the
"chromium" master, and uses the existing approach based on the name of
the builder to determine its platform.

R=stgao,chanli,lijeffrey,wylieb
BUG=810959

Change-Id: I723d39c7ae63181c3c357d7c7a42cd443db23ce3
Reviewed-on: https://chromium-review.googlesource.com/936784
Reviewed-by: Chan Li <chanli@chromium.org>
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>

[modify] https://crrev.com/00f8e25084e3cf7b47163b5fa5fb1ecfc5c5893d/appengine/findit/waterfall/test/waterfall_config_test.py
[modify] https://crrev.com/00f8e25084e3cf7b47163b5fa5fb1ecfc5c5893d/appengine/findit/waterfall/test/wf_testcase.py
[modify] https://crrev.com/00f8e25084e3cf7b47163b5fa5fb1ecfc5c5893d/appengine/findit/waterfall/waterfall_config.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/190e48dd49a3e3857d7e6a60e80b4d33b852b59f

commit 190e48dd49a3e3857d7e6a60e80b4d33b852b59f
Author: Roberto Carrillo <robertocn@google.com>
Date: Wed Feb 28 00:06:59 2018

[Findit] Add function to trigger buildaheads.

This new file will host functions related to build-aheads.

The first function added here triggers a full build in the given bot
in the cache matching the given waterfall builder.

R=stgao,chanli,lijeffrey,wylieb
BUG=810959

Change-Id: Ic40a5dbf78c37c6479cd1b3cca98fba569012ca9
Reviewed-on: https://chromium-review.googlesource.com/935443
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>
Reviewed-by: Chan Li <chanli@chromium.org>

[modify] https://crrev.com/190e48dd49a3e3857d7e6a60e80b4d33b852b59f/appengine/findit/waterfall/test/wf_testcase.py
[add] https://crrev.com/190e48dd49a3e3857d7e6a60e80b4d33b852b59f/appengine/findit/services/build_ahead.py
[add] https://crrev.com/190e48dd49a3e3857d7e6a60e80b4d33b852b59f/appengine/findit/services/test/build_ahead_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/d583b428c048dd0572363247e0a0ef8261dbe7c8

commit d583b428c048dd0572363247e0a0ef8261dbe7c8
Author: Roberto Carrillo <robertocn@google.com>
Date: Wed Feb 28 00:46:38 2018

[Findit] Add function to check the status of the tree.

This function looks at the latest status of the chromium tree, and
returns true if it can determine that the tree is open, false otherwise.

Note that this doesn't have too much in common with the functionality in
handlers/collect_tree_closures and trying to extract the common parts of
the code, may add unnecessary complexity.

R=stgao,chanli,lijeffrey,wylieb
BUG=810959

Change-Id: Ida23e0b79b253b2404f3d6741b7aa10c24fc434f
Reviewed-on: https://chromium-review.googlesource.com/939658
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>
Reviewed-by: Chan Li <chanli@chromium.org>

[modify] https://crrev.com/d583b428c048dd0572363247e0a0ef8261dbe7c8/appengine/findit/services/build_ahead.py
[modify] https://crrev.com/d583b428c048dd0572363247e0a0ef8261dbe7c8/appengine/findit/services/test/build_ahead_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/a5d9aa4575df359d2ea6d4dec2241b9f810e436a

commit a5d9aa4575df359d2ea6d4dec2241b9f810e436a
Author: Roberto Carrillo <robertocn@google.com>
Date: Wed Feb 28 01:55:00 2018

[Findit] New model to track build ahead jobs.

Note that these jobs are not associated to any failure in particular,
hence no master/builder/buildnumer composite key.

The use case is the following:
The upcoming cron job to schedule build aheads will first check for
ongoing build-aheads, and for that, it needs to query the datastore for
those entries that are still in progress, check with buildbot if they
are done, and if they are, update their status and metadata, and update
the cache statistics.

R=stgao,chanli,lijeffrey,wylieb
BUG=810959

Change-Id: I188d3a899883578c002d8b9a13a4a8bfe591c4b4
Reviewed-on: https://chromium-review.googlesource.com/938972
Reviewed-by: Chan Li <chanli@chromium.org>
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>

[add] https://crrev.com/a5d9aa4575df359d2ea6d4dec2241b9f810e436a/appengine/findit/model/build_ahead_try_job.py
[add] https://crrev.com/a5d9aa4575df359d2ea6d4dec2241b9f810e436a/appengine/findit/model/test/build_ahead_try_job_test.py
[modify] https://crrev.com/a5d9aa4575df359d2ea6d4dec2241b9f810e436a/appengine/findit/index.yaml

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/0d0cb5f405be0ff2d5bfff5afb4dc3b7fad72eb6

commit 0d0cb5f405be0ff2d5bfff5afb4dc3b7fad72eb6
Author: Roberto Carrillo <robertocn@google.com>
Date: Wed Feb 28 20:26:22 2018

[Findit] Extend WfTryBot/WfTryBotCache model to record full build stats.

Build-aheads require us to know the last revision fully built on a given
cache/bot combination.

This change also allows us to track the caches on any given bot so that
we detect cache evictions and increase a monarch metric when that
happens.

R=stgao,chanli,lijeffrey,
BUG=810959

Change-Id: Ie156ef406b131ca8ceb94d833e6c796c236e0091
Reviewed-on: https://chromium-review.googlesource.com/936782
Reviewed-by: Chan Li <chanli@chromium.org>
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>

[modify] https://crrev.com/0d0cb5f405be0ff2d5bfff5afb4dc3b7fad72eb6/appengine/findit/model/wf_try_bot_cache.py
[modify] https://crrev.com/0d0cb5f405be0ff2d5bfff5afb4dc3b7fad72eb6/appengine/findit/common/monitoring.py
[modify] https://crrev.com/0d0cb5f405be0ff2d5bfff5afb4dc3b7fad72eb6/appengine/findit/model/test/wf_try_bot_cache_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/67299b0450e42393d1689e5821206ee44c8cf486

commit 67299b0450e42393d1689e5821206ee44c8cf486
Author: Roberto Carrillo <robertocn@google.com>
Date: Thu Mar 01 00:31:52 2018

[Findit] Function to update the status of build aheads in progress.

This function updates the build ahead try job entities with the
appropriate running status and buildbucket response.

R=stgao,chanli,lijeffrey,wylieb
BUG=810959

Change-Id: Ic99dab007c787a15855deb57108cd458c6a85061
Reviewed-on: https://chromium-review.googlesource.com/939765
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>
Reviewed-by: Chan Li <chanli@chromium.org>

[modify] https://crrev.com/67299b0450e42393d1689e5821206ee44c8cf486/appengine/findit/services/build_ahead.py
[modify] https://crrev.com/67299b0450e42393d1689e5821206ee44c8cf486/appengine/findit/services/test/build_ahead_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/c0efca4891e2115d9eb09c94baefc4c38abe184c

commit c0efca4891e2115d9eb09c94baefc4c38abe184c
Author: Roberto Carrillo <robertocn@google.com>
Date: Thu Mar 01 19:59:36 2018

[Findit] Platforms to build.

This function decides which platforms a run of the build aheads cron job
should schedule for building.

The function uses as inputs the available bots for each platform, the
number of ongoing buildaheads, and the level of activity in the repo.

This is meant to select a subset of [android, mac, unix, windows] and
the cron job will later decide for a given platform which specific cache
within each select platform to trigger a build for.

R=chanli,stgao,lijeffrey,wylieb
BUG=810959

Change-Id: I52ed1faf9a137c29a3b84ca9b2e483b22f80f11a
Reviewed-on: https://chromium-review.googlesource.com/941508
Reviewed-by: Chan Li <chanli@chromium.org>
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>

[modify] https://crrev.com/c0efca4891e2115d9eb09c94baefc4c38abe184c/appengine/findit/services/build_ahead.py
[modify] https://crrev.com/c0efca4891e2115d9eb09c94baefc4c38abe184c/appengine/findit/services/test/build_ahead_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/f386fa94d8fec20c450e7f44e43af274bce85f9d

commit f386fa94d8fec20c450e7f44e43af274bce85f9d
Author: Roberto Carrillo <robertocn@google.com>
Date: Thu Mar 01 21:48:06 2018

[Findit] Pick random builder for build aheads.

The design of the build aheads feature asks for randomly choosing the
builder to trigger a build ahead for, but giving more weight for caches
that are more outdated.

R=stgao,chanli,lijeffrey,wylieb
BUG=810959

Change-Id: I58f1d8e9c4e12590fdb6f7185be60e7ce29a4939
Reviewed-on: https://chromium-review.googlesource.com/941986
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>
Reviewed-by: Chan Li <chanli@chromium.org>

[modify] https://crrev.com/f386fa94d8fec20c450e7f44e43af274bce85f9d/appengine/findit/services/build_ahead.py
[modify] https://crrev.com/f386fa94d8fec20c450e7f44e43af274bce85f9d/appengine/findit/services/test/build_ahead_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/e28662d9d6c5e53efc46c1a795bd4f51aeaddc08

commit e28662d9d6c5e53efc46c1a795bd4f51aeaddc08
Author: Roberto Carrillo <robertocn@google.com>
Date: Mon Mar 12 22:47:53 2018

[Findit] Cron job to build caches ahead of need.

This change implements the logic whereby a new cron job:
  - Updates the status of any ongoing build-aheads.
  - Determines which platforms to trigger rebuilds for based on bot
  usage.
  - Triggers build aheads until no more build aheads should be done
  (i.e. at most 50% of the capacity is used by build aheads if there's
  low repo activity, or 1 bot per platform is used if there's high
  activity, or no capacity is used if the tree is closed.)

R=stgao,chanli,lijeffrey,wylieb
BUG=810959

Change-Id: I9152ae58c02092bd6a5304336b0fd70209b96aed
Reviewed-on: https://chromium-review.googlesource.com/949417
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>
Reviewed-by: Shuotao Gao <stgao@chromium.org>
Reviewed-by: Chan Li <chanli@chromium.org>

[modify] https://crrev.com/e28662d9d6c5e53efc46c1a795bd4f51aeaddc08/appengine/findit/cron.yaml
[modify] https://crrev.com/e28662d9d6c5e53efc46c1a795bd4f51aeaddc08/appengine/findit/services/test/build_ahead_test.py
[add] https://crrev.com/e28662d9d6c5e53efc46c1a795bd4f51aeaddc08/appengine/findit/handlers/build_ahead.py
[modify] https://crrev.com/e28662d9d6c5e53efc46c1a795bd4f51aeaddc08/appengine/findit/services/build_ahead.py
[add] https://crrev.com/e28662d9d6c5e53efc46c1a795bd4f51aeaddc08/appengine/findit/handlers/test/build_ahead_test.py
[modify] https://crrev.com/e28662d9d6c5e53efc46c1a795bd4f51aeaddc08/appengine/findit/main.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/e9a52f2c5a560206d3c09c5a9a9abe2f95bc1ff1

commit e9a52f2c5a560206d3c09c5a9a9abe2f95bc1ff1
Author: Roberto Carrillo <robertocn@google.com>
Date: Tue Jun 26 18:30:02 2018

[Findit] Enable build-aheads during business hours.

R=stgao,chanli

Bug: 810959
Change-Id: I1e58e48d7e3f8e3395cf77bc4ca59ed539223897
Reviewed-on: https://chromium-review.googlesource.com/1114230
Reviewed-by: Shuotao Gao <stgao@chromium.org>
Commit-Queue: Roberto Carrillo <robertocn@chromium.org>

[modify] https://crrev.com/e9a52f2c5a560206d3c09c5a9a9abe2f95bc1ff1/appengine/findit/cron.yaml