New issue
Advanced search Search tips

Issue 764043 link

Starred by 2 users
Status: Fixed
Owner:
vadimsh@chromium.org
Closed: May 2018
Cc:
tandrii@chromium.org
no...@chromium.org
vadimsh@chromium.org
Components:
EstimatedDays: 7
NextAction: ----
OS: ----
Pri: 2
Type: Feature

Blocking:
issue 805625



Sign in to add a comment

Implement Scheduler v2

Project Member Reported by vadimsh@chromium.org, Sep 11 2017 
In which each invocation is its own entity group. This will allow to run multiple invocations in parallel, and will remove 1 QPS limit on rate of changes of a job (and its invocations). This is important for overall stability of high rate jobs.
Project Member

Comment 1 by bugdroid1@chromium.org, Sep 12 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/af2196e28a0b79b66902a31e6b9f75caf525eb25

commit af2196e28a0b79b66902a31e6b9f75caf525eb25
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Sep 12 00:20:56 2017

scheduler: Extract the transaction helper function.

Make it retry on subset of transient errors. Consistently mark all unexpected
datastore errors (usually deadlines) as transient, to make sure their status
correctly propagates to the task queue level.

R=tandrii@chromium.org
BUG= 764043 ,  691399 

Change-Id: I31c289f12eb91cf932022bc85b9c7f7368309fe6
Reviewed-on: https://chromium-review.googlesource.com/660987
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/af2196e28a0b79b66902a31e6b9f75caf525eb25/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/af2196e28a0b79b66902a31e6b9f75caf525eb25/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/af2196e28a0b79b66902a31e6b9f75caf525eb25/scheduler/appengine/engine/invocation.go
[modify] https://crrev.com/af2196e28a0b79b66902a31e6b9f75caf525eb25/scheduler/appengine/engine/utils.go
[add] https://crrev.com/af2196e28a0b79b66902a31e6b9f75caf525eb25/scheduler/appengine/engine/utils_test.go
Project Member

Comment 2 by bugdroid1@chromium.org, Sep 12 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/a4aeb12108039006ec4386edcc48089d1051eaeb

commit a4aeb12108039006ec4386edcc48089d1051eaeb
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Sep 12 00:31:36 2017

scheduler: Rename txn(...) to jobTxn(...) to clarify its meaning.

This will remove some ambiguity when we have transactions around Invocation
entity groups (when they are separate from Job entity groups).

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I57c53ff76e01d97f9525856793ed5050f7f69698
Reviewed-on: https://chromium-review.googlesource.com/660989
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/a4aeb12108039006ec4386edcc48089d1051eaeb/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/a4aeb12108039006ec4386edcc48089d1051eaeb/scheduler/appengine/engine/engine.go
Project Member

Comment 3 by bugdroid1@chromium.org, Sep 12 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/8c2c2b158d6ea761fc142194808beee8de757702

commit 8c2c2b158d6ea761fc142194808beee8de757702
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Sep 12 00:38:26 2017

scheduler: Pass jobID explicitly from controller instead of through JobKey.

In the future (when invocations are their own entity groups) Invocation.JobKey
won't be set.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: Ic2558652df36d6561ff4c61bf40c392492590055
Reviewed-on: https://chromium-review.googlesource.com/660991
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/8c2c2b158d6ea761fc142194808beee8de757702/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/8c2c2b158d6ea761fc142194808beee8de757702/scheduler/appengine/engine/engine.go
Project Member

Comment 4 by bugdroid1@chromium.org, Sep 12 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/ed7fee7abddfd32591eb64e5728a92e84d94ff97

commit ed7fee7abddfd32591eb64e5728a92e84d94ff97
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Sep 12 17:20:36 2017

scheduler: Add StatusRetrying invocation state that indicates a retry.

This allows to clean up retry logic a little bit in preparation for v2 schema,
where retries will reuse existing Invocation entity instead of creating a new
one.

Also fix devcfg configs used when running stuff locally.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I1de40235cd9e5094e59fe973a7e2114d85f7c463
Reviewed-on: https://chromium-review.googlesource.com/662397
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/ed7fee7abddfd32591eb64e5728a92e84d94ff97/scheduler/appengine/devcfg/projects/project1/luci-scheduler-dev.cfg
[modify] https://crrev.com/ed7fee7abddfd32591eb64e5728a92e84d94ff97/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/ed7fee7abddfd32591eb64e5728a92e84d94ff97/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/ed7fee7abddfd32591eb64e5728a92e84d94ff97/scheduler/appengine/engine/engine_test.go
[modify] https://crrev.com/ed7fee7abddfd32591eb64e5728a92e84d94ff97/scheduler/appengine/task/task.go
[modify] https://crrev.com/ed7fee7abddfd32591eb64e5728a92e84d94ff97/scheduler/appengine/ui/presentation.go
Project Member

Comment 5 by bugdroid1@chromium.org, Sep 13 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/9ddaedf1519998c33926b974c5a2142089a03e47

commit 9ddaedf1519998c33926b974c5a2142089a03e47
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Sep 13 00:05:37 2017

scheduler: Extract Job entity update into a separate function.

This decouples the controller from Job entity, allowing the controller to be
reused by v2 engine.

No functional changes, pure refactoring.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: Idb0d28179f34a513ec0d163522d47c0fb01dd7d0
Reviewed-on: https://chromium-review.googlesource.com/663893
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/9ddaedf1519998c33926b974c5a2142089a03e47/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/9ddaedf1519998c33926b974c5a2142089a03e47/scheduler/appengine/engine/engine.go
Project Member

Comment 6 by bugdroid1@chromium.org, Sep 13 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/90631bee7c94129a4e25f74656774b86ac3cefa7

commit 90631bee7c94129a4e25f74656774b86ac3cefa7
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Sep 13 01:31:47 2017

scheduler: Extract controller instantiation into withController function.

This removes some repeated code. Introduces slight behavior change: previously
a transient error in HandleTimer or HandleNotification was logged inside
invocation debug log, now it is only logged in GAE log (and Invocation entity is
not updated at all). This is slightly cleaner, since previously we also could
potentially instantiate more stuff (like new timers/trigger) when saving the 
debug log. Now we do a clean retry from scratch instead.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I257387ee03626f88497db5b10ec80e5762f7c16c
Reviewed-on: https://chromium-review.googlesource.com/664224
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/90631bee7c94129a4e25f74656774b86ac3cefa7/scheduler/appengine/engine/engine.go
Project Member

Comment 7 by bugdroid1@chromium.org, Sep 13 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/18260b33b709ea3485d1c9e48eb9a31345e9c9d6

commit 18260b33b709ea3485d1c9e48eb9a31345e9c9d6
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Sep 13 01:53:59 2017

scheduler: Add jobID argument to GetVisibleInvocationsByNonce.

This slightly simplifies switching to v2 jobs since we'll know in advance what
kind of invocations to expect from the query (v1 or v2), based on the known
job ID.

This is also slightly safer, since we now guarantee that caller sees only
invocations of the job they requested. Before there was (very small) chance
to encounter a collision in nonces and see someone else's invocations.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I14bba2ad8124e313b3be352db23672a6b3978004
Reviewed-on: https://chromium-review.googlesource.com/664180
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/18260b33b709ea3485d1c9e48eb9a31345e9c9d6/scheduler/appengine/apiservers/scheduler_test.go
[modify] https://crrev.com/18260b33b709ea3485d1c9e48eb9a31345e9c9d6/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/18260b33b709ea3485d1c9e48eb9a31345e9c9d6/scheduler/appengine/engine/engine_test.go
[modify] https://crrev.com/18260b33b709ea3485d1c9e48eb9a31345e9c9d6/scheduler/appengine/ui/job.go
Project Member

Comment 8 by bugdroid1@chromium.org, Sep 13 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/cbacdbae8693ee5df766eff72d8228ac95a118bf

commit cbacdbae8693ee5df766eff72d8228ac95a118bf
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Sep 13 19:29:20 2017

scheduler: Add a notion of v2 Jobs and Invocations.

In v2 schema Invocations are their own entity groups that reference parent jobs
via JobID indexed field (and not via JobKey parent entity key). This requires
some adjustment into how such invocations are created and queried.

v2 are not used anywhere yet at all, thus there's no tests changes. Tests
confirm that v1 still works fine.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I29fe768df8cb26170e1821af343a0499c4a8118b
Reviewed-on: https://chromium-review.googlesource.com/663794
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/cbacdbae8693ee5df766eff72d8228ac95a118bf/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/cbacdbae8693ee5df766eff72d8228ac95a118bf/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/cbacdbae8693ee5df766eff72d8228ac95a118bf/scheduler/appengine/engine/invocation.go
[modify] https://crrev.com/cbacdbae8693ee5df766eff72d8228ac95a118bf/scheduler/appengine/engine/utils.go
Project Member

Comment 10 by bugdroid1@chromium.org, Sep 13 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/73f977f6709f815f5984983f9a78f1c963ee68e8

commit 73f977f6709f815f5984983f9a78f1c963ee68e8
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Sep 13 22:56:50 2017

scheduler: Don't embed Config, convert it to a regular field.

Embedding is not really appropriate here, since engineImpl is not a "subtype"
of Config.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I3c2198a61a7b32a91ed19cc8116f57f123e98bad
Reviewed-on: https://chromium-review.googlesource.com/666225
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/73f977f6709f815f5984983f9a78f1c963ee68e8/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/73f977f6709f815f5984983f9a78f1c963ee68e8/scheduler/appengine/engine/engine.go
Project Member

Comment 12 by bugdroid1@chromium.org, Sep 14 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/db27840137914bd387bd40f87d049f66ff7e9bbf

commit db27840137914bd387bd40f87d049f66ff7e9bbf
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Sep 14 23:13:17 2017

scheduler: Reshuffle engine.go methods to be less chaotic.

Also cleanup comments and do minor renaming for consistency. No actual code
changes.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: Icc08c0f44a6c31d661269999c2cdc5234dcc5810
Reviewed-on: https://chromium-review.googlesource.com/668082
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/db27840137914bd387bd40f87d049f66ff7e9bbf/scheduler/appengine/apiservers/scheduler_test.go
[modify] https://crrev.com/db27840137914bd387bd40f87d049f66ff7e9bbf/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/db27840137914bd387bd40f87d049f66ff7e9bbf/scheduler/appengine/ui/job.go
Project Member

Comment 15 by bugdroid1@chromium.org, Sep 15 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/f307b7bf4ba6d6eb07c20a0e532e82ea33ca4543

commit f307b7bf4ba6d6eb07c20a0e532e82ea33ca4543
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Sep 15 04:21:08 2017

scheduler: Fix "context" import.

TBR=tandrii@chromium.org
BUG= 764043 

Change-Id: I9689f23024871a216c7059fcf8ee36a488808968
Reviewed-on: https://chromium-review.googlesource.com/668126
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/f307b7bf4ba6d6eb07c20a0e532e82ea33ca4543/scheduler/appengine/engine/job.go
Project Member

Comment 17 by bugdroid1@chromium.org, Sep 16 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/87935867649f02f65778dcc493fe50ae060375b3

commit 87935867649f02f65778dcc493fe50ae060375b3
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Sat Sep 16 00:45:58 2017

scheduler: Make some more code in engine.go reusable by v2 engine.

Basically, we are extracting v1 vs v2 differences into a separate class called
jobController.

Pure refactoring, no functionality changes (and thus no unit test changes).

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I1ae750651f838961aa1871339c31eb16d0da00db
Reviewed-on: https://chromium-review.googlesource.com/668237
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/87935867649f02f65778dcc493fe50ae060375b3/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/87935867649f02f65778dcc493fe50ae060375b3/scheduler/appengine/engine/engine.go
[add] https://crrev.com/87935867649f02f65778dcc493fe50ae060375b3/scheduler/appengine/engine/engine_v1.go
[add] https://crrev.com/87935867649f02f65778dcc493fe50ae060375b3/scheduler/appengine/engine/engine_v2.go
Project Member

Comment 19 by bugdroid1@chromium.org, Sep 16 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/f667ef426dd8d85849a090b017a138ee43532c2f

commit f667ef426dd8d85849a090b017a138ee43532c2f
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Sat Sep 16 01:26:18 2017

scheduler: Index only finished invocations in v2.

This is actually not used yet. See the comment for IndexedJobID field for the
rationale.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: Ibb9e7d584c6f254194dd0a0c1e41c87499f889d8
Reviewed-on: https://chromium-review.googlesource.com/669692
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/f667ef426dd8d85849a090b017a138ee43532c2f/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/f667ef426dd8d85849a090b017a138ee43532c2f/scheduler/appengine/engine/invocation.go
Project Member

Comment 20 by bugdroid1@chromium.org, Sep 16 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/d14dff80d3442d069b570c094a3a96d0e184827f

commit d14dff80d3442d069b570c094a3a96d0e184827f
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Sat Sep 16 02:56:18 2017

scheduler: Store triggers in the Invocation entity.

Display them in UI while at it.

R=tandrii@chromium.org
BUG= 764043 , 646067

Change-Id: I64bf9f4c669fe1224d8f231cb4f792b2e7e920c4
Reviewed-on: https://chromium-review.googlesource.com/670099
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/d14dff80d3442d069b570c094a3a96d0e184827f/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/d14dff80d3442d069b570c094a3a96d0e184827f/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/d14dff80d3442d069b570c094a3a96d0e184827f/scheduler/appengine/engine/invocation.go
[modify] https://crrev.com/d14dff80d3442d069b570c094a3a96d0e184827f/scheduler/appengine/engine/statemachine.go
[modify] https://crrev.com/d14dff80d3442d069b570c094a3a96d0e184827f/scheduler/appengine/engine/utils.go
[modify] https://crrev.com/d14dff80d3442d069b570c094a3a96d0e184827f/scheduler/appengine/frontend/templates/pages/invocation.html
[modify] https://crrev.com/d14dff80d3442d069b570c094a3a96d0e184827f/scheduler/appengine/task/gitiles/gitiles.go
[modify] https://crrev.com/d14dff80d3442d069b570c094a3a96d0e184827f/scheduler/appengine/task/task.go
[modify] https://crrev.com/d14dff80d3442d069b570c094a3a96d0e184827f/scheduler/appengine/ui/presentation.go
Project Member

Comment 21 by bugdroid1@chromium.org, Sep 18 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/99acfe4926e1765157c839790f532758dc6408dc

commit 99acfe4926e1765157c839790f532758dc6408dc
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Mon Sep 18 21:57:26 2017

scheduler: Fix panic in recordOverrun.

"invalid input type (**engine.Invocation): not a PLS, pointer-to-struct, or
slice thereof"

Cased by recent refactoring that changed Invocation to *Invocation. Datastore's
Put accepts interface{} and it didn't notice the change.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I2d8a5015680703e6cc4bbffb327832031233a5b5
Reviewed-on: https://chromium-review.googlesource.com/670074
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/99acfe4926e1765157c839790f532758dc6408dc/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/99acfe4926e1765157c839790f532758dc6408dc/scheduler/appengine/engine/engine_test.go
Project Member

Comment 22 by bugdroid1@chromium.org, Sep 30 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/5e968a5ca706627c12702efa9d90e638bd0defdc

commit 5e968a5ca706627c12702efa9d90e638bd0defdc
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Sat Sep 30 03:06:39 2017

tq: Add a testing helper.

It's a layer on top of raw taskqueue.Testable interface that helps with
deserializing and routing "fake" test tasks.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I8fdb3a72f110123830945ecd3f492e5c4d84419c
Reviewed-on: https://chromium-review.googlesource.com/691286
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[add] https://crrev.com/5e968a5ca706627c12702efa9d90e638bd0defdc/appengine/tq/internals.go
[modify] https://crrev.com/5e968a5ca706627c12702efa9d90e638bd0defdc/appengine/tq/tq.go
[add] https://crrev.com/5e968a5ca706627c12702efa9d90e638bd0defdc/appengine/tq/tqtesting/testable.go
[add] https://crrev.com/5e968a5ca706627c12702efa9d90e638bd0defdc/appengine/tq/tqtesting/testable_test.go
Project Member

Comment 23 by bugdroid1@chromium.org, Oct 3 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/28d4e01a524404a445aa8c7e9b34575f18d531a0

commit 28d4e01a524404a445aa8c7e9b34575f18d531a0
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Oct 03 00:15:25 2017

scheduler: Implement 'ForceInvocation' using v2 data structures.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: Ib65c8961bccbed81a34c4c08b142741a2c328e41
Reviewed-on: https://chromium-review.googlesource.com/671852
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/28d4e01a524404a445aa8c7e9b34575f18d531a0/scheduler/appengine/devcfg/projects/project1/luci-scheduler-dev.cfg
[modify] https://crrev.com/28d4e01a524404a445aa8c7e9b34575f18d531a0/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/28d4e01a524404a445aa8c7e9b34575f18d531a0/scheduler/appengine/engine/engine_test.go
[modify] https://crrev.com/28d4e01a524404a445aa8c7e9b34575f18d531a0/scheduler/appengine/engine/engine_v2.go
[add] https://crrev.com/28d4e01a524404a445aa8c7e9b34575f18d531a0/scheduler/appengine/engine/engine_v2_test.go
[modify] https://crrev.com/28d4e01a524404a445aa8c7e9b34575f18d531a0/scheduler/appengine/engine/invocation.go
[modify] https://crrev.com/28d4e01a524404a445aa8c7e9b34575f18d531a0/scheduler/appengine/engine/job.go
[add] https://crrev.com/28d4e01a524404a445aa8c7e9b34575f18d531a0/scheduler/appengine/engine/triage.go
[add] https://crrev.com/28d4e01a524404a445aa8c7e9b34575f18d531a0/scheduler/appengine/engine/triage_test.go
[modify] https://crrev.com/28d4e01a524404a445aa8c7e9b34575f18d531a0/scheduler/appengine/engine/utils.go
[modify] https://crrev.com/28d4e01a524404a445aa8c7e9b34575f18d531a0/scheduler/appengine/frontend/queue.yaml
[modify] https://crrev.com/28d4e01a524404a445aa8c7e9b34575f18d531a0/scheduler/appengine/internal/tq.pb.go
[modify] https://crrev.com/28d4e01a524404a445aa8c7e9b34575f18d531a0/scheduler/appengine/internal/tq.proto
[modify] https://crrev.com/28d4e01a524404a445aa8c7e9b34575f18d531a0/scheduler/appengine/ui/job.go

Comment 26 by efoo@chromium.org, Oct 3 2017

Labels: LUCI-M3-S9 REQBY-LUCI-M4-OpenBeta LUCI-M3-ClosedBeta
Vadim, can you add in estimated days remaining time for this? Thanks!
Project Member

Comment 27 by bugdroid1@chromium.org, Oct 3 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/04a6c8e626611c1dea65ada103dbf30e6cdc31e6

commit 04a6c8e626611c1dea65ada103dbf30e6cdc31e6
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Oct 03 22:23:15 2017

scheduler: Replace task.Trigger with proto message.

This reduces number of deserializations we have to make in various places
(but adds them to a bunch of different places, in particular in v1 engine that
prefers JSON over proto for internal tasks).

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I8a7389f7ffa730f7275c0c779d485ca851c5fa99
Reviewed-on: https://chromium-review.googlesource.com/695952
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/catalog/catalog_test.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/engine/engine_test.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/engine/engine_v1.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/engine/engine_v2.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/engine/engine_v2_test.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/engine/invocation.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/engine/statemachine.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/engine/statemachine_test.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/engine/utils.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/engine/utils_test.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/internal/tq.pb.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/internal/triggers.pb.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/internal/triggers.proto
[delete] https://crrev.com/8456783da2352d3b7343fde44f44ceef01d52f30/scheduler/appengine/internal/types.pb.go
[delete] https://crrev.com/8456783da2352d3b7343fde44f44ceef01d52f30/scheduler/appengine/internal/types.proto
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/task/buildbucket/buildbucket.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/task/buildbucket/buildbucket_test.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/task/gitiles/gitiles.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/task/gitiles/gitiles_test.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/task/noop/noop.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/task/noop/noop_test.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/task/swarming/swarming.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/task/task.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/task/urlfetch/urlfetch.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/task/utils/tasktest/controller.go
[modify] https://crrev.com/04a6c8e626611c1dea65ada103dbf30e6cdc31e6/scheduler/appengine/ui/presentation.go

Comment 28 by vadimsh@chromium.org, Oct 4 2017

EstimatedDays: 7
Project Member

Comment 29 by bugdroid1@chromium.org, Oct 5 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/875ac0c0f2f756adf3bf1e9f6a4dc91bc5f65524

commit 875ac0c0f2f756adf3bf1e9f6a4dc91bc5f65524
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Oct 05 18:57:32 2017

tq: Add ability to stop testing simulation before particular tasks.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I722782e03d5c5bfea7ac073c26ea8aa0026c6c69
Reviewed-on: https://chromium-review.googlesource.com/702020
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/875ac0c0f2f756adf3bf1e9f6a4dc91bc5f65524/appengine/tq/tqtesting/testable.go
[modify] https://crrev.com/875ac0c0f2f756adf3bf1e9f6a4dc91bc5f65524/appengine/tq/tqtesting/testable_test.go
Project Member

Comment 30 by bugdroid1@chromium.org, Oct 5 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/35655b2c8c9d026519e8fec793669ec318ae0152

commit 35655b2c8c9d026519e8fec793669ec318ae0152
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Oct 05 19:04:02 2017

tq: Add TaskList.Payloads() method that gets a slice of payloads.

Very handy when writing asserts for an array of tasks.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I6bdd03f6df01b144a9506ed73d6b3c5a83f96686
Reviewed-on: https://chromium-review.googlesource.com/701977
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/35655b2c8c9d026519e8fec793669ec318ae0152/appengine/tq/tqtesting/testable.go
[modify] https://crrev.com/35655b2c8c9d026519e8fec793669ec318ae0152/appengine/tq/tqtesting/testable_test.go

Comment 31 by vadimsh@chromium.org, Oct 5 2017 

What's left to get feature parity with v1:

1. Migrate "invocation timers" to v2.
2. Migrate cron logic to v2 (including overruns recording).
3. Implement pause/resume and enable/disable.
4. Adjust UI and API to display active invocations and states of v2 jobs.
5. Implement automatic migration of jobs from v1 to v2.
6. Perform this migration.
7. Kill old code.

After that we can easily start adding new feature that weren't possible with v1 (like "tree stability timer" or multiple concurrent builds).
Project Member

Comment 32 by bugdroid1@chromium.org, Oct 5 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/ab15e560b46c8af262a76221c5553469a4a4095e

commit ab15e560b46c8af262a76221c5553469a4a4095e
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Oct 05 23:07:24 2017

scheduler: Implement triggers in v2.

Very basic for now. More features will be added after the cron part is
integrated.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: Id26f22c3fc958510423ba2f430e7f9672cb34884
Reviewed-on: https://chromium-review.googlesource.com/696269
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/ab15e560b46c8af262a76221c5553469a4a4095e/scheduler/appengine/devcfg/projects/project1/luci-scheduler-dev.cfg
[modify] https://crrev.com/ab15e560b46c8af262a76221c5553469a4a4095e/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/ab15e560b46c8af262a76221c5553469a4a4095e/scheduler/appengine/engine/engine_v2.go
[modify] https://crrev.com/ab15e560b46c8af262a76221c5553469a4a4095e/scheduler/appengine/engine/engine_v2_test.go
[modify] https://crrev.com/ab15e560b46c8af262a76221c5553469a4a4095e/scheduler/appengine/engine/job.go
[modify] https://crrev.com/ab15e560b46c8af262a76221c5553469a4a4095e/scheduler/appengine/engine/triage.go
[modify] https://crrev.com/ab15e560b46c8af262a76221c5553469a4a4095e/scheduler/appengine/engine/triage_test.go
[modify] https://crrev.com/ab15e560b46c8af262a76221c5553469a4a4095e/scheduler/appengine/frontend/queue.yaml
[modify] https://crrev.com/ab15e560b46c8af262a76221c5553469a4a4095e/scheduler/appengine/internal/tq.pb.go
[modify] https://crrev.com/ab15e560b46c8af262a76221c5553469a4a4095e/scheduler/appengine/internal/tq.proto

Comment 33 by efoo@chromium.org, Oct 17 2017

Labels: LUCI-M3-S10

Comment 34 by efoo@chromium.org, Oct 17 2017

Labels: LUCI-Blocker-M4
Project Member

Comment 35 by bugdroid1@chromium.org, Nov 7 2017 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/17fa674d2ba90791320f24c8a161adf42eff2816

commit 17fa674d2ba90791320f24c8a161adf42eff2816
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Nov 07 03:07:27 2017

scheduler: Implement invocation timers in v2.

They are now also stored inside Invocation entities, which makes them somewhat
similar to triggers. It improves handling of timer tick retries and adds more
visibility into what's going on.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: Iacd9643e609b7a0c82c00db66eddd31db7e1c0ab
Reviewed-on: https://chromium-review.googlesource.com/704264
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/17fa674d2ba90791320f24c8a161adf42eff2816/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/17fa674d2ba90791320f24c8a161adf42eff2816/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/17fa674d2ba90791320f24c8a161adf42eff2816/scheduler/appengine/engine/engine_v1.go
[modify] https://crrev.com/17fa674d2ba90791320f24c8a161adf42eff2816/scheduler/appengine/engine/engine_v2.go
[modify] https://crrev.com/17fa674d2ba90791320f24c8a161adf42eff2816/scheduler/appengine/engine/engine_v2_test.go
[modify] https://crrev.com/17fa674d2ba90791320f24c8a161adf42eff2816/scheduler/appengine/engine/invocation.go
[modify] https://crrev.com/17fa674d2ba90791320f24c8a161adf42eff2816/scheduler/appengine/engine/utils.go
[modify] https://crrev.com/17fa674d2ba90791320f24c8a161adf42eff2816/scheduler/appengine/frontend/queue.yaml
[add] https://crrev.com/17fa674d2ba90791320f24c8a161adf42eff2816/scheduler/appengine/internal/timers.pb.go
[add] https://crrev.com/17fa674d2ba90791320f24c8a161adf42eff2816/scheduler/appengine/internal/timers.proto
[modify] https://crrev.com/17fa674d2ba90791320f24c8a161adf42eff2816/scheduler/appengine/internal/tq.pb.go
[modify] https://crrev.com/17fa674d2ba90791320f24c8a161adf42eff2816/scheduler/appengine/internal/tq.proto
[modify] https://crrev.com/17fa674d2ba90791320f24c8a161adf42eff2816/scheduler/appengine/internal/triggers.pb.go
[modify] https://crrev.com/17fa674d2ba90791320f24c8a161adf42eff2816/scheduler/appengine/task/task.go

Comment 36 by efoo@chromium.org, Nov 7 2017

Labels: -REQBY-LUCI-M4-OpenBeta -LUCI-M3-ClosedBeta REQBY-LUCI-M4a-MacWinPOC LUCI-M3-Beta LUCI-M3-S11

Comment 37 by efoo@chromium.org, Nov 16 2017

Labels: LUCI-M3-S12

Comment 38 by efoo@chromium.org, Dec 5 2017

Labels: LUCI-M3-S13

Comment 39 by efoo@chromium.org, Dec 19 2017

Labels: -LUCI-M3-Beta -LUCI-M3-S13 LUCI-M0-Backlog

Comment 40 by efoo@chromium.org, Dec 20 2017

Labels: -REQBY-LUCI-M4a-MacWinPOC REQBY-LUCI-M5-GASupport

Comment 42 by vadimsh@chromium.org, Jan 24 2018

Blocking: 805625
Project Member

Comment 43 by bugdroid1@chromium.org, Jan 25 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/d8ec75d36afb4290d44739384c622e19729fb3dd

commit d8ec75d36afb4290d44739384c622e19729fb3dd
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Jan 25 23:42:41 2018

[scheduler] Rename InvocationRequest to task.Request and move to Controller.

Pure refactoring, no logic changes.

This is in preparation for adding move stuff (such as a build properties and
tags, derived from triggers) to the request.

R=tandrii@chromium.org, nodir@chromium.org
BUG= 794499 ,  764043 

Change-Id: I7bd35cc4b6cd631ce9ea31fda205fced880f5cce
Reviewed-on: https://chromium-review.googlesource.com/885507
Reviewed-by: Erik Staab <estaab@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/catalog/catalog_test.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/engine/engine_test.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/engine/engine_v2.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/engine/engine_v2_test.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/engine/invocation.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/engine/triage.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/engine/triage_test.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/buildbucket/buildbucket.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/buildbucket/buildbucket_test.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/gitiles/gitiles.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/gitiles/gitiles_test.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/noop/noop.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/noop/noop_test.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/task.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/urlfetch/urlfetch.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/urlfetch/urlfetch_test.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/utils/tasktest/controller.go

Comment 44 by efoo@chromium.org, Jan 25 2018

Labels: luci-blocker-migration
Project Member

Comment 45 by bugdroid1@chromium.org, Jan 26 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/2337be80bf794c93576604258f0c8d1e70eade4f

commit 2337be80bf794c93576604258f0c8d1e70eade4f
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Jan 26 00:01:01 2018

[scheduler] Move protos with trigger bodies from internal into API.

We'll add new BuildbucketTrigger proto there and then will use them in new
EmitTriggers RPC.

Keep the full-fledged Trigger (the one containing all metadata) in the internal
still, there's no need to expose it to the outside world for now.

Also drop "Data" suffix in the proto message names, it is redundant now.

R=tandrii@chromium.org, nodir@chromium.org
BUG= 794499 ,  764043 

Change-Id: Iba1508d71cacf3627cf43f01994466191d26dc85
Reviewed-on: https://chromium-review.googlesource.com/885623
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/api/scheduler/v1/pb.discovery.go
[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/api/scheduler/v1/scheduler.pb.go
[add] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/api/scheduler/v1/triggers.pb.go
[add] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/api/scheduler/v1/triggers.proto
[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/appengine/engine/engine_test.go
[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/appengine/internal/timers.pb.go
[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/appengine/internal/triggers.pb.go
[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/appengine/internal/triggers.proto
[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/appengine/task/buildbucket/buildbucket.go
[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/appengine/task/buildbucket/buildbucket_test.go
[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/appengine/task/gitiles/gitiles.go
[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/appengine/task/gitiles/gitiles_test.go
Project Member

Comment 46 by bugdroid1@chromium.org, Jan 26 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/9647fda714506eeb471444c921b5ba7f6a7425c7

commit 9647fda714506eeb471444c921b5ba7f6a7425c7
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Jan 26 19:57:37 2018

[scheduler] Allow to order triggers by time they were emitted.

This is needed to sort triggers (which are stored as unordered sets in v2
version of the engine) when there's no better comparator available.

Be careful to handle "emitted at exact same moment in time" situation correctly
by using tuple (timestamp, order in batch).

This CL populates the fields, but doesn't use them yet.

R=tandrii@chromium.org
BUG= 794499 ,  764043 

Change-Id: I5a25955eacddc10a2bf0a0b69ddb530eaeb2dda2
Reviewed-on: https://chromium-review.googlesource.com/888175
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/9647fda714506eeb471444c921b5ba7f6a7425c7/scheduler/api/scheduler/v1/pb.discovery.go
[modify] https://crrev.com/9647fda714506eeb471444c921b5ba7f6a7425c7/scheduler/api/scheduler/v1/scheduler.pb.go
[modify] https://crrev.com/9647fda714506eeb471444c921b5ba7f6a7425c7/scheduler/api/scheduler/v1/scheduler.proto
[modify] https://crrev.com/9647fda714506eeb471444c921b5ba7f6a7425c7/scheduler/appengine/apiservers/scheduler.go
[modify] https://crrev.com/9647fda714506eeb471444c921b5ba7f6a7425c7/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/9647fda714506eeb471444c921b5ba7f6a7425c7/scheduler/appengine/engine/engine_v2_test.go
[modify] https://crrev.com/9647fda714506eeb471444c921b5ba7f6a7425c7/scheduler/appengine/internal/triggers.pb.go
[modify] https://crrev.com/9647fda714506eeb471444c921b5ba7f6a7425c7/scheduler/appengine/internal/triggers.proto

Comment 47 by efoo@chromium.org, Jan 30 2018

Labels: -LUCI-Blocker-M4

Comment 48 by efoo@chromium.org, Jan 30 2018

Labels: LUCI-Backlog

Comment 49 by efoo@chromium.org, Jan 30 2018

Labels: -LUCI-M0-Backlog

Comment 50 by efoo@chromium.org, Feb 13 2018

Labels: -LUCI-blocker-migration LUCI-Chromium LUCI-Blocker-Chromium

Comment 51 by efoo@chromium.org, Feb 13 2018

Labels: LUCI-Chromium-CQSets

Comment 52 by efoo@chromium.org, Feb 13 2018

Labels: -LUCI-Chromium-CQSets

Comment 53 by efoo@chromium.org, Feb 15 2018

Labels: -REQBY-LUCI-M5-GASupport
Project Member

Comment 54 by bugdroid1@chromium.org, Apr 7 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/39f709bbe485b86f4d43a81f2cf1c214bff4d8e0

commit 39f709bbe485b86f4d43a81f2cf1c214bff4d8e0
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Sat Apr 07 02:06:38 2018

[scheduler] Add Generation counter and Equal() method to cron.State.

Generation counter will be used to generate IDs of triggers emitted by the cron
(for deduplication), and Equal() is needed in Job.Equal() implementation.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: Ie30b63c7fd93e82b0cab127633073016fde661d4
Reviewed-on: https://chromium-review.googlesource.com/1000420
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/39f709bbe485b86f4d43a81f2cf1c214bff4d8e0/scheduler/appengine/engine/cron/machine.go
[modify] https://crrev.com/39f709bbe485b86f4d43a81f2cf1c214bff4d8e0/scheduler/appengine/engine/cron/machine_test.go
Project Member

Comment 55 by bugdroid1@chromium.org, Apr 9 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/814c5b671f3d213f482ebf29560421c91fc44ccb

commit 814c5b671f3d213f482ebf29560421c91fc44ccb
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Mon Apr 09 04:29:48 2018

[scheduler] Add CronTrigger trigger type.

It is a special internally used trigger that will be generated by the cron
state machine. External API users can see it, but can't emit it.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I438b36eac9fa9487db4046cb8031b5b505613bbf
Reviewed-on: https://chromium-review.googlesource.com/1000428
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/814c5b671f3d213f482ebf29560421c91fc44ccb/scheduler/api/scheduler/v1/pb.discovery.go
[modify] https://crrev.com/814c5b671f3d213f482ebf29560421c91fc44ccb/scheduler/api/scheduler/v1/scheduler.pb.go
[modify] https://crrev.com/814c5b671f3d213f482ebf29560421c91fc44ccb/scheduler/api/scheduler/v1/triggers.pb.go
[modify] https://crrev.com/814c5b671f3d213f482ebf29560421c91fc44ccb/scheduler/api/scheduler/v1/triggers.proto
[modify] https://crrev.com/814c5b671f3d213f482ebf29560421c91fc44ccb/scheduler/appengine/apiservers/scheduler.go
[modify] https://crrev.com/814c5b671f3d213f482ebf29560421c91fc44ccb/scheduler/appengine/engine/request.go
[modify] https://crrev.com/814c5b671f3d213f482ebf29560421c91fc44ccb/scheduler/appengine/internal/triggers.pb.go
[modify] https://crrev.com/814c5b671f3d213f482ebf29560421c91fc44ccb/scheduler/appengine/internal/triggers.proto
Project Member

Comment 56 by bugdroid1@chromium.org, Apr 10 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/a5ec678434d70d371b3412aa9e8385185b3b24fd

commit a5ec678434d70d371b3412aa9e8385185b3b24fd
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Apr 10 00:25:04 2018

[tq] Add ShouldStopAfter predicate to the simulator.

It'll make some scheduler tests clearer.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I3c8697f04e16b016f56080acdcf4d4cee7c9f86e
Reviewed-on: https://chromium-review.googlesource.com/1003634
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/a5ec678434d70d371b3412aa9e8385185b3b24fd/appengine/tq/tqtesting/testable.go
[modify] https://crrev.com/a5ec678434d70d371b3412aa9e8385185b3b24fd/appengine/tq/tqtesting/testable_test.go
Project Member

Comment 57 by bugdroid1@chromium.org, Apr 10 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/f665464d87d9c7c84d3dcb6958a15dfba5cb39e5

commit f665464d87d9c7c84d3dcb6958a15dfba5cb39e5
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Apr 10 00:34:25 2018

[scheduler] Implement basic cron scheduling in v2.

Each job now owns a cron state machine which emits CronTriggers for this job
based on the configured schedule. This trigger is then triaged as usual.

To support "with 10min interval" schedules, the cron machine is being rewound
during the triage if there's no invocations running anymore.

Things not supported yet:
  * Overrun status is not displayed. In fact, the new behavior is "if a job is
    running at the time new invocation should be starting, queue up the new
    invocation and launch it as soon as the previous one finishes" (this is just
    default behavior of the triggers queue).
  * Job status in UI is not displayed correctly, including "Next run" field.

This is largely based on the old prototype:
https://github.com/luci/luci-go/blob/1b403a7ea/scheduler/appengine/engine/cron/demo/main.go

R=tandrii@chromium.org
BUG= 764043 

Change-Id: Icc48936917e58fb9aacb1e5a4150f0deaec65444
Reviewed-on: https://chromium-review.googlesource.com/1000431
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[add] https://crrev.com/f665464d87d9c7c84d3dcb6958a15dfba5cb39e5/scheduler/appengine/engine/cron.go
[modify] https://crrev.com/f665464d87d9c7c84d3dcb6958a15dfba5cb39e5/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/f665464d87d9c7c84d3dcb6958a15dfba5cb39e5/scheduler/appengine/engine/engine_v1.go
[modify] https://crrev.com/f665464d87d9c7c84d3dcb6958a15dfba5cb39e5/scheduler/appengine/engine/engine_v2.go
[modify] https://crrev.com/f665464d87d9c7c84d3dcb6958a15dfba5cb39e5/scheduler/appengine/engine/engine_v2_test.go
[modify] https://crrev.com/f665464d87d9c7c84d3dcb6958a15dfba5cb39e5/scheduler/appengine/engine/job.go
[modify] https://crrev.com/f665464d87d9c7c84d3dcb6958a15dfba5cb39e5/scheduler/appengine/engine/triage.go
[modify] https://crrev.com/f665464d87d9c7c84d3dcb6958a15dfba5cb39e5/scheduler/appengine/frontend/queue.yaml
[modify] https://crrev.com/f665464d87d9c7c84d3dcb6958a15dfba5cb39e5/scheduler/appengine/internal/timers.pb.go
[modify] https://crrev.com/f665464d87d9c7c84d3dcb6958a15dfba5cb39e5/scheduler/appengine/internal/tq.pb.go
[modify] https://crrev.com/f665464d87d9c7c84d3dcb6958a15dfba5cb39e5/scheduler/appengine/internal/tq.proto
Project Member

Comment 58 by bugdroid1@chromium.org, Apr 10 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/535c3726bbdf94b14b02174d29365ccd8af24b76

commit 535c3726bbdf94b14b02174d29365ccd8af24b76
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Apr 10 00:41:54 2018

[scheduler] Add KickTriageTask that transactionally invokes a triage.

This will be useful later when implementing Pause/Resume for v2 jobs.

We can't transactionally enqueue TriageJobStateTask, since its throttling
mechanism uses memcache and named tasks, which are not available inside
transactions. So instead transactions can enqueue KickTriageTask, which in
turn will enqueue TriageJobStateTask (with throttling).

To avoid confusing method naming (is 'kickTriageTask' some task handler or
initiator?), prefix all task handler methods with 'exec'.

And finally rename kickTriageJobStateTask to just kickTriageNow, so we now have
two alternatives to launch a triage: kickTriageNow and kickTriageLater.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I2be881c39544053e63e144ecbbe96f84c2617c02
Reviewed-on: https://chromium-review.googlesource.com/1003317
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/535c3726bbdf94b14b02174d29365ccd8af24b76/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/535c3726bbdf94b14b02174d29365ccd8af24b76/scheduler/appengine/engine/engine_v2_test.go
[modify] https://crrev.com/535c3726bbdf94b14b02174d29365ccd8af24b76/scheduler/appengine/engine/utils.go
[modify] https://crrev.com/535c3726bbdf94b14b02174d29365ccd8af24b76/scheduler/appengine/internal/timers.pb.go
[modify] https://crrev.com/535c3726bbdf94b14b02174d29365ccd8af24b76/scheduler/appengine/internal/tq.pb.go
[modify] https://crrev.com/535c3726bbdf94b14b02174d29365ccd8af24b76/scheduler/appengine/internal/tq.proto

Comment 59 by vadimsh@chromium.org, Apr 10 2018

Summary: Implement Scheduler v2 (was: Implement Scheduler's v2 datastore schema)
Project Member

Comment 61 by bugdroid1@chromium.org, Apr 10 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/ecee9b800a3cc257280d3760602eda901b7b4bf8

commit ecee9b800a3cc257280d3760602eda901b7b4bf8
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Apr 10 06:36:20 2018

[scheduler] Implement EmitTriggers RPC for v2 jobs.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: Ibaf6123e80cd110cd79becd5cc6018658ea2cd63
Reviewed-on: https://chromium-review.googlesource.com/1004115
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/ecee9b800a3cc257280d3760602eda901b7b4bf8/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/ecee9b800a3cc257280d3760602eda901b7b4bf8/scheduler/appengine/engine/engine_v2_test.go
Project Member

Comment 63 by bugdroid1@chromium.org, Apr 11 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/66d7c3c493dcd7ce1fa1897512501378432164f9

commit 66d7c3c493dcd7ce1fa1897512501378432164f9
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Apr 11 02:30:54 2018

[scheduler] Store list of recently finished invocations ID in Job entity.

This would allow to build almost perfectly consistent listing of latest
invocations. Almost, because we will still rely on datastore indexes to catch up
within FinishedInvocationsHorizon (which is 10 min).

At least there'll be no gaps in the listing for most of day-to-day usage.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I842dbf90b754d5f813e3221344a84024984d99c8
Reviewed-on: https://chromium-review.googlesource.com/1006417
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/66d7c3c493dcd7ce1fa1897512501378432164f9/scheduler/appengine/engine/job.go
[modify] https://crrev.com/66d7c3c493dcd7ce1fa1897512501378432164f9/scheduler/appengine/engine/triage.go
[modify] https://crrev.com/66d7c3c493dcd7ce1fa1897512501378432164f9/scheduler/appengine/engine/triage_test.go
[modify] https://crrev.com/66d7c3c493dcd7ce1fa1897512501378432164f9/scheduler/appengine/engine/utils.go
[add] https://crrev.com/66d7c3c493dcd7ce1fa1897512501378432164f9/scheduler/appengine/internal/db.pb.go
[add] https://crrev.com/66d7c3c493dcd7ce1fa1897512501378432164f9/scheduler/appengine/internal/db.proto
[modify] https://crrev.com/66d7c3c493dcd7ce1fa1897512501378432164f9/scheduler/appengine/internal/timers.pb.go
[modify] https://crrev.com/66d7c3c493dcd7ce1fa1897512501378432164f9/scheduler/appengine/internal/tq.pb.go
[modify] https://crrev.com/66d7c3c493dcd7ce1fa1897512501378432164f9/scheduler/appengine/internal/triggers.pb.go
Project Member

Comment 64 by bugdroid1@chromium.org, Apr 12 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/367481c892669432db74b667d568b764484a1992

commit 367481c892669432db74b667d568b764484a1992
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Apr 12 01:18:56 2018

[scheduler] Add a stub for getting active invocations, use it from UI.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: Ie40820b52aad0fae7e7028f1c9f9a91538356071
Reviewed-on: https://chromium-review.googlesource.com/1008068
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/367481c892669432db74b667d568b764484a1992/scheduler/appengine/apiservers/scheduler.go
[modify] https://crrev.com/367481c892669432db74b667d568b764484a1992/scheduler/appengine/apiservers/scheduler_test.go
[modify] https://crrev.com/367481c892669432db74b667d568b764484a1992/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/367481c892669432db74b667d568b764484a1992/scheduler/appengine/engine/engine_test.go
[modify] https://crrev.com/367481c892669432db74b667d568b764484a1992/scheduler/appengine/engine/engine_v2_test.go
[modify] https://crrev.com/367481c892669432db74b667d568b764484a1992/scheduler/appengine/frontend/templates/pages/job.html
[modify] https://crrev.com/367481c892669432db74b667d568b764484a1992/scheduler/appengine/ui/common.go
[modify] https://crrev.com/367481c892669432db74b667d568b764484a1992/scheduler/appengine/ui/job.go

Comment 65 by sergeybe...@chromium.org, Apr 12 2018 

Issue 640821 has been merged into this issue.
Project Member

Comment 66 by bugdroid1@chromium.org, Apr 16 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/09a08ebe50a853573e779b9566da1cfbab9915b1

commit 09a08ebe50a853573e779b9566da1cfbab9915b1
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Mon Apr 16 19:14:15 2018

[scheduler] Add a proto to represent pagination cursor.

For now it just wraps datastore cursor, but soon it will have more fields
needed for paginating over active invocations list.

R=tandrii@chromium.org, jchinlee@chromium.org
BUG= 764043 

Change-Id: I9e5cda68068398e39f17b6de158ba94eaf5deefd
Reviewed-on: https://chromium-review.googlesource.com/1012991
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[add] https://crrev.com/09a08ebe50a853573e779b9566da1cfbab9915b1/scheduler/appengine/engine/cursors.go
[modify] https://crrev.com/09a08ebe50a853573e779b9566da1cfbab9915b1/scheduler/appengine/engine/engine.go
[add] https://crrev.com/09a08ebe50a853573e779b9566da1cfbab9915b1/scheduler/appengine/internal/cursors.pb.go
[add] https://crrev.com/09a08ebe50a853573e779b9566da1cfbab9915b1/scheduler/appengine/internal/cursors.proto
[modify] https://crrev.com/09a08ebe50a853573e779b9566da1cfbab9915b1/scheduler/appengine/internal/db.pb.go
[modify] https://crrev.com/09a08ebe50a853573e779b9566da1cfbab9915b1/scheduler/appengine/internal/timers.pb.go
[modify] https://crrev.com/09a08ebe50a853573e779b9566da1cfbab9915b1/scheduler/appengine/internal/tq.pb.go
[modify] https://crrev.com/09a08ebe50a853573e779b9566da1cfbab9915b1/scheduler/appengine/internal/triggers.pb.go
Project Member

Comment 67 by bugdroid1@chromium.org, Apr 18 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/f051ce1d158a2193cc131ae3e1829eca3f838f43

commit f051ce1d158a2193cc131ae3e1829eca3f838f43
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Apr 18 02:24:12 2018

[scheduler] Add a helper that converts a datastore query into an iterator.

It is simpler to merge results of multiple queries of different nature when
they are iterators. Callbacks (as used by datastore.Run function) aren't
composable.

R=tandrii@chromium.org, jchinlee@chromium.org
BUG= 764043 

Change-Id: I1b2d94b4e35329e507783333e452e6a08b9013c9
Reviewed-on: https://chromium-review.googlesource.com/1014649
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[add] https://crrev.com/f051ce1d158a2193cc131ae3e1829eca3f838f43/scheduler/appengine/engine/invquery.go
[add] https://crrev.com/f051ce1d158a2193cc131ae3e1829eca3f838f43/scheduler/appengine/engine/invquery_test.go
Project Member

Comment 69 by bugdroid1@chromium.org, Apr 21 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/396dd838c89c4cd1edb45e451bc3a5c0960675d4

commit 396dd838c89c4cd1edb45e451bc3a5c0960675d4
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Sat Apr 21 00:12:40 2018

[scheduler] Improve consistency of "Running" state indicator.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I3447b01326ea87aedd870f3c682fd550acccf762
Reviewed-on: https://chromium-review.googlesource.com/1023063
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/396dd838c89c4cd1edb45e451bc3a5c0960675d4/scheduler/appengine/ui/job.go
Project Member

Comment 71 by bugdroid1@chromium.org, Apr 24 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/bc7f1cb7f6ed7fe81eadda61f06f81f29921badc

commit bc7f1cb7f6ed7fe81eadda61f06f81f29921badc
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Apr 24 01:18:09 2018

[scheduler] Double save finished v1 invocations as v2.

To start populating history of v2 invocations before the full switch. Better to
have at least something there.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: Iae1ddfa665a7de11d64d503bb9205a452338505b
Reviewed-on: https://chromium-review.googlesource.com/1024721
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/bc7f1cb7f6ed7fe81eadda61f06f81f29921badc/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/bc7f1cb7f6ed7fe81eadda61f06f81f29921badc/scheduler/appengine/engine/engine_test.go
[modify] https://crrev.com/bc7f1cb7f6ed7fe81eadda61f06f81f29921badc/scheduler/appengine/engine/invocation.go
Project Member

Comment 72 by bugdroid1@chromium.org, Apr 24 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/26138910505fd36f4601af9bc7004a53e56a6cee

commit 26138910505fd36f4601af9bc7004a53e56a6cee
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Apr 24 02:25:59 2018

[scheduler] Allow v1 jobs to trigger v2 jobs.

This situation will be possible during the migration to v2 and must work.
Similarly v2 jobs should be able to trigger v1 jobs, this will be done in the
next CL.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I33bbaf715bf3219ff7c7bccbbf8b399a1a75a0de
Reviewed-on: https://chromium-review.googlesource.com/1024867
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/26138910505fd36f4601af9bc7004a53e56a6cee/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/26138910505fd36f4601af9bc7004a53e56a6cee/scheduler/appengine/engine/engine_test.go
Project Member

Comment 73 by bugdroid1@chromium.org, Apr 24 2018

Labels: merge-merged-config
The following revision refers to this bug:
  https://chromium.googlesource.com/infra/experimental/+/4b70a417b988cece67b937326b3498b09464c432

commit 4b70a417b988cece67b937326b3498b09464c432
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Apr 24 03:35:32 2018

[luci-scheduler-dev] Test v1 triggering v2 and vice versa.

TBR=tandrii@chromium.org
BUG= 764043 

Change-Id: I1466d9d15d4c8dd8e33e77d9c02985db8509a269
Reviewed-on: https://chromium-review.googlesource.com/1025406
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/4b70a417b988cece67b937326b3498b09464c432/luci-scheduler-dev.cfg
Project Member

Comment 74 by bugdroid1@chromium.org, Apr 24 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/f1b8bc903589d7384b0b24dd945158cab08d9f27

commit f1b8bc903589d7384b0b24dd945158cab08d9f27
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Apr 24 16:44:47 2018

[tq] Add 'UnknownTaskHandler' callback to RunSimulation.

It can be used to handle all tasks unrecognized by tq.Dispatcher. Will be useful
for testing that Scheduler v2 (that uses tq.Dispatcher generally) still
correctly emits TQ tasks for v1 jobs (that use raw taskqueue API).

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I7f140e894af00781a226ffff2e03bd36e76d41a7
Reviewed-on: https://chromium-review.googlesource.com/1025404
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/f1b8bc903589d7384b0b24dd945158cab08d9f27/appengine/tq/tqtesting/testable.go
[modify] https://crrev.com/f1b8bc903589d7384b0b24dd945158cab08d9f27/appengine/tq/tqtesting/testable_test.go
Project Member

Comment 75 by bugdroid1@chromium.org, Apr 24 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/5d0aeb37e5886681cc04ae872517557bc4fa9744

commit 5d0aeb37e5886681cc04ae872517557bc4fa9744
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Apr 24 19:21:59 2018

[scheduler] Allow v2 jobs to trigger v1 jobs.

This situation will be possible during the migration to v2 and must work.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I9e91801624c1b692dd5cec75c9c1275fd31097d4
Reviewed-on: https://chromium-review.googlesource.com/1025405
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/5d0aeb37e5886681cc04ae872517557bc4fa9744/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/5d0aeb37e5886681cc04ae872517557bc4fa9744/scheduler/appengine/engine/engine_v2_test.go
Project Member

Comment 76 by bugdroid1@chromium.org, Apr 25 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/4947c581e3558ffb329af2f62e53b0e5c65136ca

commit 4947c581e3558ffb329af2f62e53b0e5c65136ca
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Apr 25 21:56:38 2018

[scheduler] Implement AbortJob in v2.

It just calls AbortInvocation for all active invocations.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: Ic9d2981b99f36359713ac69a5979ad6df83d0c75
Reviewed-on: https://chromium-review.googlesource.com/1026976
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/4947c581e3558ffb329af2f62e53b0e5c65136ca/scheduler/appengine/engine/engine_v2.go
[modify] https://crrev.com/4947c581e3558ffb329af2f62e53b0e5c65136ca/scheduler/appengine/engine/engine_v2_test.go
Project Member

Comment 77 by bugdroid1@chromium.org, Apr 25 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/507cedbf25f8b4a16c778f7ab62f99e062af2464

commit 507cedbf25f8b4a16c778f7ab62f99e062af2464
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Apr 25 22:00:08 2018

[scheduler] Add a mechanism to opt in jobs to v1->v2 migration.

Since doing everything at once seems risky. It is temporary, and thus quite
crude.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: Ife2e57cb93c333fd324d085448d44d52fb37bb9c
Reviewed-on: https://chromium-review.googlesource.com/1027200
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/507cedbf25f8b4a16c778f7ab62f99e062af2464/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/507cedbf25f8b4a16c778f7ab62f99e062af2464/scheduler/appengine/engine/job.go
[add] https://crrev.com/507cedbf25f8b4a16c778f7ab62f99e062af2464/scheduler/appengine/engine/migration/settings.go
Project Member

Comment 78 by bugdroid1@chromium.org, Apr 25 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/d4f8764f0c5d20786da0bd9d4b4f2cb715d68afd

commit d4f8764f0c5d20786da0bd9d4b4f2cb715d68afd
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Apr 25 23:22:48 2018

[scheduler] Gracefully handle "late" v1 TQ tasks for v2 jobs.

They are possible right after the job has been converted to v2.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I07dd8baad87b8615bffa2a0a8ad0495712c90847
Reviewed-on: https://chromium-review.googlesource.com/1029001
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/d4f8764f0c5d20786da0bd9d4b4f2cb715d68afd/scheduler/appengine/engine/engine.go
Project Member

Comment 79 by bugdroid1@chromium.org, Apr 26 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/experimental/+/eb609d5b2be104e8724caa9b8e4ede795c05f3c6

commit eb609d5b2be104e8724caa9b8e4ede795c05f3c6
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Apr 26 05:15:48 2018

[luci-scheduler-dev] Add various test cases for v1->v2 migration.

TBR=tandrii@chromium.org
BUG= 764043 

Change-Id: I4265def036d70893299d9ec8b92c651194174fe2
Reviewed-on: https://chromium-review.googlesource.com/1029618
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/eb609d5b2be104e8724caa9b8e4ede795c05f3c6/luci-scheduler-dev.cfg
Project Member

Comment 80 by bugdroid1@chromium.org, Apr 26 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/1e04250311d6fa7001d9ea7dcf5aaad9702333bb

commit 1e04250311d6fa7001d9ea7dcf5aaad9702333bb
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Apr 26 17:41:30 2018

[scheduler] Implement v1 -> v2 conversion.

We introduce a new terminal state to v1 state machine called CONVERTING_TO_V2.
The v1 state machine switches to it when it finds itself in-between invocations.
While doing so, it emits ConvertToV2Action that finishes the transition to v2,
rescheduling the cron tick with v2 engine and transferring the pending triggers.

To avoid waiting for rarely changing jobs (e.g. crons that tick once a week),
we also proactively nudge v1 state machines in UpdateProjectJobs if they appear
in-between invocations.

No unit tests, since this code is going to be deleted ASAP, and it was manually
tested on luci-scheduler-dev under various conditions.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: Iff534893a602c73fbdf100c5b5908b08c8bb7b06
Reviewed-on: https://chromium-review.googlesource.com/1029614
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/1e04250311d6fa7001d9ea7dcf5aaad9702333bb/scheduler/appengine/engine/cron/machine.go
[modify] https://crrev.com/1e04250311d6fa7001d9ea7dcf5aaad9702333bb/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/1e04250311d6fa7001d9ea7dcf5aaad9702333bb/scheduler/appengine/engine/statemachine.go
[modify] https://crrev.com/1e04250311d6fa7001d9ea7dcf5aaad9702333bb/scheduler/appengine/engine/utils.go
Project Member

Comment 81 by bugdroid1@chromium.org, Apr 27 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/3975dbf8ff1561a4e1bc13e98d895809d39ff489

commit 3975dbf8ff1561a4e1bc13e98d895809d39ff489
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Apr 27 23:55:18 2018

[scheduler] Merge engine_test.go and engine_test_v2.go and delete test for v1.

It is somewhat reduces the code coverage. Missing tests will be readded once v1
is fully gone.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I65265784a18d673d50da2d2ec015d5584f4693da
Reviewed-on: https://chromium-review.googlesource.com/1034016
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/3975dbf8ff1561a4e1bc13e98d895809d39ff489/scheduler/appengine/engine/engine_test.go
[delete] https://crrev.com/ae10a60c1d343edc9a1434cfd350cbd232ca365f/scheduler/appengine/engine/engine_v2_test.go
[add] https://crrev.com/3975dbf8ff1561a4e1bc13e98d895809d39ff489/scheduler/appengine/engine/helpers_test.go
Project Member

Comment 82 by bugdroid1@chromium.org, Apr 28 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/cb2bb36f52415dec987c515222da20d2ab5b057b

commit cb2bb36f52415dec987c515222da20d2ab5b057b
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Sat Apr 28 21:56:07 2018

[scheduler] Remove job.IsV2() and inv.IsV2() and immediate dependencies.

All jobs and invocations are V2 now.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I143e9eafedaae19ee6e84056711801e0913da78f
Reviewed-on: https://chromium-review.googlesource.com/1034124
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/cb2bb36f52415dec987c515222da20d2ab5b057b/scheduler/api/scheduler/v1/pb.discovery.go
[modify] https://crrev.com/cb2bb36f52415dec987c515222da20d2ab5b057b/scheduler/api/scheduler/v1/scheduler.pb.go
[modify] https://crrev.com/cb2bb36f52415dec987c515222da20d2ab5b057b/scheduler/api/scheduler/v1/scheduler.proto
[modify] https://crrev.com/cb2bb36f52415dec987c515222da20d2ab5b057b/scheduler/appengine/apiservers/scheduler_test.go
[modify] https://crrev.com/cb2bb36f52415dec987c515222da20d2ab5b057b/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/cb2bb36f52415dec987c515222da20d2ab5b057b/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/cb2bb36f52415dec987c515222da20d2ab5b057b/scheduler/appengine/engine/engine_test.go
[delete] https://crrev.com/3975dbf8ff1561a4e1bc13e98d895809d39ff489/scheduler/appengine/engine/engine_v1.go
[modify] https://crrev.com/cb2bb36f52415dec987c515222da20d2ab5b057b/scheduler/appengine/engine/invocation.go
[modify] https://crrev.com/cb2bb36f52415dec987c515222da20d2ab5b057b/scheduler/appengine/engine/invquery.go
[modify] https://crrev.com/cb2bb36f52415dec987c515222da20d2ab5b057b/scheduler/appengine/engine/invquery_test.go
[modify] https://crrev.com/cb2bb36f52415dec987c515222da20d2ab5b057b/scheduler/appengine/engine/job.go
[delete] https://crrev.com/3975dbf8ff1561a4e1bc13e98d895809d39ff489/scheduler/appengine/engine/migration/settings.go
[modify] https://crrev.com/cb2bb36f52415dec987c515222da20d2ab5b057b/scheduler/appengine/frontend/templates/includes/base.html
[modify] https://crrev.com/cb2bb36f52415dec987c515222da20d2ab5b057b/scheduler/appengine/frontend/templates/pages/index.html
[modify] https://crrev.com/cb2bb36f52415dec987c515222da20d2ab5b057b/scheduler/appengine/frontend/templates/pages/project.html
[modify] https://crrev.com/cb2bb36f52415dec987c515222da20d2ab5b057b/scheduler/appengine/presentation/state.go
[modify] https://crrev.com/cb2bb36f52415dec987c515222da20d2ab5b057b/scheduler/appengine/presentation/state_test.go
[modify] https://crrev.com/cb2bb36f52415dec987c515222da20d2ab5b057b/scheduler/appengine/ui/job.go
[modify] https://crrev.com/cb2bb36f52415dec987c515222da20d2ab5b057b/scheduler/appengine/ui/presentation.go
Project Member

Comment 86 by bugdroid1@chromium.org, Apr 30 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/3641f4b08b8b9c32ffec3e6a7bbe92d4ad9f467a

commit 3641f4b08b8b9c32ffec3e6a7bbe92d4ad9f467a
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Mon Apr 30 18:15:55 2018

[scheduler] Remove InvocationNonce, it is no longer needed.

In v2 we use InvocationID itself for the same purpose.

Also do some tiny collateral fixes:
  * Clearer debug message for new starting invocation.
  * Correctly populate noop trigger body in noop task.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: I4ba753853cd6dea194bfe086dea2f3dafdca3a1a
Reviewed-on: https://chromium-review.googlesource.com/1034127
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/3641f4b08b8b9c32ffec3e6a7bbe92d4ad9f467a/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/3641f4b08b8b9c32ffec3e6a7bbe92d4ad9f467a/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/3641f4b08b8b9c32ffec3e6a7bbe92d4ad9f467a/scheduler/appengine/engine/engine_test.go
[modify] https://crrev.com/3641f4b08b8b9c32ffec3e6a7bbe92d4ad9f467a/scheduler/appengine/engine/invocation.go
[modify] https://crrev.com/3641f4b08b8b9c32ffec3e6a7bbe92d4ad9f467a/scheduler/appengine/task/buildbucket/buildbucket.go
[modify] https://crrev.com/3641f4b08b8b9c32ffec3e6a7bbe92d4ad9f467a/scheduler/appengine/task/noop/noop.go
[modify] https://crrev.com/3641f4b08b8b9c32ffec3e6a7bbe92d4ad9f467a/scheduler/appengine/task/noop/noop_test.go
[modify] https://crrev.com/3641f4b08b8b9c32ffec3e6a7bbe92d4ad9f467a/scheduler/appengine/task/task.go
[modify] https://crrev.com/3641f4b08b8b9c32ffec3e6a7bbe92d4ad9f467a/scheduler/appengine/task/utils/tasktest/controller.go
Project Member

Comment 87 by bugdroid1@chromium.org, Apr 30 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/b99764eeb8355e4c8f3a6766fc54e1253e4b8658

commit b99764eeb8355e4c8f3a6766fc54e1253e4b8658
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Mon Apr 30 18:22:15 2018

[scheduler] Remove jobController and associated abstractions.

They are not needed anymore. Also do the final clean up pass to make sure all
v1 stuff has been removed, all comments are up-to-date, etc.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: Ib51277d51460db3b50bbaf9904aeb9fabafdbc3f
Reviewed-on: https://chromium-review.googlesource.com/1034160
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/b99764eeb8355e4c8f3a6766fc54e1253e4b8658/scheduler/api/scheduler/v1/pb.discovery.go
[modify] https://crrev.com/b99764eeb8355e4c8f3a6766fc54e1253e4b8658/scheduler/api/scheduler/v1/scheduler.pb.go
[modify] https://crrev.com/b99764eeb8355e4c8f3a6766fc54e1253e4b8658/scheduler/api/scheduler/v1/scheduler.proto
[modify] https://crrev.com/b99764eeb8355e4c8f3a6766fc54e1253e4b8658/scheduler/appengine/apiservers/scheduler_test.go
[modify] https://crrev.com/b99764eeb8355e4c8f3a6766fc54e1253e4b8658/scheduler/appengine/devcfg/projects/project1/luci-scheduler-dev.cfg
[modify] https://crrev.com/b99764eeb8355e4c8f3a6766fc54e1253e4b8658/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/b99764eeb8355e4c8f3a6766fc54e1253e4b8658/scheduler/appengine/engine/cron.go
[modify] https://crrev.com/b99764eeb8355e4c8f3a6766fc54e1253e4b8658/scheduler/appengine/engine/cron/machine.go
[modify] https://crrev.com/b99764eeb8355e4c8f3a6766fc54e1253e4b8658/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/b99764eeb8355e4c8f3a6766fc54e1253e4b8658/scheduler/appengine/engine/engine_test.go
[delete] https://crrev.com/3641f4b08b8b9c32ffec3e6a7bbe92d4ad9f467a/scheduler/appengine/engine/engine_v2.go
[modify] https://crrev.com/b99764eeb8355e4c8f3a6766fc54e1253e4b8658/scheduler/appengine/engine/utils.go
[modify] https://crrev.com/b99764eeb8355e4c8f3a6766fc54e1253e4b8658/scheduler/appengine/ui/job.go
Project Member

Comment 88 by bugdroid1@chromium.org, May 1 2018 

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/1884fe9fa0fe5ec320563d70240cc81ff4d61e45

commit 1884fe9fa0fe5ec320563d70240cc81ff4d61e45
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue May 01 19:32:24 2018

[scheduler] Readd some tests that were removed during v1->v2 migration.

R=tandrii@chromium.org
BUG= 764043 

Change-Id: Id56e5da805028208562c9d39959e0a7218e7f411
Reviewed-on: https://chromium-review.googlesource.com/1036225
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/1884fe9fa0fe5ec320563d70240cc81ff4d61e45/scheduler/appengine/engine/engine_test.go
[modify] https://crrev.com/1884fe9fa0fe5ec320563d70240cc81ff4d61e45/scheduler/appengine/engine/helpers_test.go

Comment 89 by vadimsh@chromium.org, May 1 2018

Status: Fixed (was: Assigned)
I consider this done. Adding new features on top of the new v2 structure is separate work.

Comment 90 by no...@chromium.org, May 1 2018 

Awesome!
►

Sign in to add a comment