This is blocking V8's migration to LUCI.

Thanks for filing. I was aware and it is on my/vadimsh@ plate, but it's good to have a bug on file.

This will wait for scheduler v2.

machenbach@ was correct - this is already biting us in case of mac debug tests, which run considerably longer than builder:

Deleted: TgTOD3OwCWB.png 283 KB

	TgTOD3OwCWB.png 283 KB View Download

link to waterfall: https://ci.chromium.org/p/chromium/g/migration-side-by-side/console

What's the proposed long-term solution? Does scheduler send a build request to buildbucket per revision? Would we need to have buildbucket merge them?

Please see issue 640821 and the referenced doc go/waterfall-scheduler-doc . I'm not sure why luci-scheduler didn't implement batching from the get go (I'm sure there were good reasons, I'm just not aware of them), but long term IMHO such scheduler-based batching could simplify many things and solve many of the existing problems, including this bug.

Merging builds in buildbucket IMNSHO is at the wrong level of abstraction.

Scheduler does batching. In fact, currently it can't do not-batching. This bugs asks for support for not-batching.

Right, but is that batching parameterized? E.g. can I say "batch at most N builds", where N can potentially be 1?

Not currently. Currently it always batches everything it can.

Marking for backlog. We will need this done during migrations but does not block task force from starting on CQ builder sets. 

note that the "1 build per job at a time" limitation does not work for "tester" builders http://g/luci-eng/MKKrbxSPDD8

Tester builders currently use buildbucket directly. So we'll either need to change buildbucket or change how builders trigger testers.

IIUC, the only reason we have builder/tester split today is because there was no other way to parallelize tests on buildbot, given its limited scheduler.

If luci-scheduler had parameterized batching (as go/waterfall-scheduler-doc suggested), we could have multiple machines in the builder pool doing building + testing-on-swarming. No need for splitting testers, and no need to change buildbucket.

If we want to compile more frequently than test (e.g. due to limited capacity for tests), one can either:
(a) setup up a separate test builder with different batch size, and run it on the latest isolate from the builder, or 
(b) trigger tests with N% probability within the same builder/tester bot, similar to how CQ triggers <100% experimental builds, using deterministic random number generator based on the build params.

>> (a) setup up a separate test builder with different batch size, and run it on the latest isolate from the builder 

This is an interesting proposal that we can be already used to solve our issues with migrating V8 CI testers to LUCI. There we actually do want to merge incoming requests, but can't do so because they are triggered via buildbucket. One issues that I see with this is determining what's the latest isolate on the tester without it having been triggered by the builder. Is there some support for this in current recipe modules or will we need to implement our own mechanism?

As for the original request, then indeed a scheduler described in go/waterfall-scheduler-doc that can schedule every Nth revision would work for us. We'll just set N to 1.

Re: #c20: I don't believe we have a mechanism right now to determine the latest isolate from a builder. It needs to be implemented.

Here are a few ideas:

1. Set up a pubsub channel from the builder that receives {isolate hash, timestamp} messages. The tester bot will run as a cron job (periodic time triggered), and will read all the messages from the pubsub channel and take the latest one, according to its timestamp. I believe luci-scheduler can skip a run if the previous job is still running.

2. Implement pubsub-based triggering in luci-scheduler, and use the same scheduling algorithm as for Gitiles (batching received messages as needed, etc.). The builder would still need to publish its isolate hashes to PubSub.

3. Implement web-hook based triggering in luci-scheduler (same batching algo applies). The builder calls a web hook in luci-scheduler with the isolate hash(es), and the scheduler decides what to do.

I'd vote for #2 or #3, as those are more generic solutions that can be reused. However, #1 is likely the fastest to implement for your specific use case.

Note that so far the mid-term (this Q) plan looks like:

1. Finish "LUCI Scheduler v2", see  Issue 764043  (in particular the design doc mentioned). This is roughly equivalent to implementing a scheduler described in go/waterfall-scheduler-doc (I think), just inside LUCI Scheduler guts (on top of datastore and task queues).

2. Teach Buildbucket builds to emit trigger events through LUCI Scheduler API. The scheduler supports triggers (that's how gitiles poller triggers builds), it just doesn't expose this functionality as external API. This is roughly equivalent to item (3) in the Sergey's list above.

3. Then we'd be able to write in luci-scheduler.cfg config somethings similar to:

job {
  id: "Linux Builder"
  
  ...

  buildbucket: {
    server: "cr-buildbucket.appspot.com"
    bucket: "luci.chromium.ci"
    builder: "Linux Builder"
  }

  batching: {
    mergeTriggers: false
  }

  triggers: "Linux Tester"
}


job {
  id: "Linux Tester"
  
  ...

  buildbucket: {
    server: "cr-buildbucket.appspot.com"
    bucket: "luci.chromium.ci"
    builder: "Linux Tester"
  }

  batching: {
    mergeTriggers: true
  }
}

(where 'batching' can have much richer structure, I just put minimal example)

---

Note that currently we have essentially "mergeTriggers: true" hardcoded in LUCI Scheduler v1 logic, but the rest of this schema applies. So with relatively little effort we can expose "EmitTrigger" API to the recipes and have testers triggered through LUCI Scheduler and auto-merged. This won't be configurable though until Scheduler v2 is finished. In particular, there'll be no way control granularity of test request merges.

implications of design in c#21:
- the graph of builders with "triggers" edges is defined statically in a config
- list of builders that a builder triggers is static, recipe cannot specify it
- when triggering, recipe cannot specify different properties for different builders. The properties it specifies are sent to all configured builders

John, would this work for Chromium?

Well, we can extend EmitTrigger(<triggered build properties>) to be EmitTrigger({job ID => <triggered build properties>}) and specify what jobs are triggered in the recipes when calling EmitTrigger(...), but this would be different from what Scheduler does now, and I'd prefer to avoid moving configuration-like data (list of triggered builds) into recipes, if possible.

So I'm actually very confused by what exactly is requested is this bug.

I think we want TESTER builds to be merged, as we do now in chromium Buildbot waterfalls. This can be done relatively easy by triggering through existing LUCI Scheduler as described in #22.

Yet the bug titles says "Allow for scheduler to not merge build requests" (i.e. complete opposite). As Andrii noted in #2, this is not doable with Scheduler v1 and requires v2 ( issue 764043 ). This is substantially more work and can't be done easy.

So what is it? I'll be working under assumption we want to merge build requests, since I believe we need it for Chromium waterfalls.

It seems to me merging of TESTER builds are needed for all bugs blocked by this one (including V8 migration, as mentioned in Sergiy's comment #20 "we actually do want to merge incoming requests"). Also this bug has all associated tracking labels, so I'll just rename it to indicate it's about MERGING build requests.

If NOT merging is also needed (it is not clear to my why it is needed other than "it is blocking v8 migration". Why is it blocking?), please file a new bug. This one unfortunately morphed into an opposite of what was initially requested 🤷

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/09b7adc3fe6d79db28af8e32fe0c2b059043c025

commit 09b7adc3fe6d79db28af8e32fe0c2b059043c025
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Wed Jan 24 01:56:09 2018

[cr-buildbucket] Return a service account email used by a build.

This will be useful in LUCI Scheduler to authorize EmitTrigger RPC calls. The
scheduler would allow calls from a service account associated with the build it
triggered.

R=nodir@chromium.org
BUG= 794499 

Change-Id: Ic84d66991769194daa1522f065c25436b4c448ac
Reviewed-on: https://chromium-review.googlesource.com/882607
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[modify] https://crrev.com/09b7adc3fe6d79db28af8e32fe0c2b059043c025/appengine/cr-buildbucket/api_common.py
[modify] https://crrev.com/09b7adc3fe6d79db28af8e32fe0c2b059043c025/appengine/cr-buildbucket/swarming/swarming.py
[modify] https://crrev.com/09b7adc3fe6d79db28af8e32fe0c2b059043c025/appengine/cr-buildbucket/swarming/test/swarming_test.py
[modify] https://crrev.com/09b7adc3fe6d79db28af8e32fe0c2b059043c025/appengine/cr-buildbucket/model.py
[modify] https://crrev.com/09b7adc3fe6d79db28af8e32fe0c2b059043c025/appengine/cr-buildbucket/test/api_common_test.py

>> If NOT merging is also needed (it is not clear to my why it is needed other than "it is blocking v8 migration". Why is it blocking?), please file a new bug. This one unfortunately morphed into an opposite of what was initially requested 🤷

Filed issue 805625.

Thanks!

>> implications of design in c#21:
>> - list of builders that a builder triggers is static, recipe cannot specify it
>> John, would this work for Chromium?

This may not work for V8. The internal v8/trigger_proxy recipe may choose not to trigger other builders in case the isolate hash has not changed from last run. For most builders that it triggers, we can use buildbucket directly because we do _not_ want them to be merged, so this is not an issue. However, one builder requires builds to be merged using logarithmic algorithm, i.e. it will merge log(N) builds, where N is the size of the pending queue. Should such functionality be implemented in luci-scheduler, we would want to be able decide whether we want to trigger a child build or not from the recipe.

In general, it seems that custom merging algorithms were used in a few places, e.g. [1], [2], [3]. It does not seems to be supported in current or proposed design of the LUCI Scheduler (go/waterfall-scheduler-doc), hence it looks like a regression in LUCI. One alternative that I was thinking about is to let users write this logic in a recipe. One would only need an API to peek at the current pending queue (should be available via buildbucket) and then decide whether to add new build request or replace an existing one with one that will combine the two. WDYT?


[1]: https://chromium.googlesource.com/chromium/tools/build/+/3414fa041df5965be8a45b7cf7266045880ea202/masters/master.client.nacl.sdk/master.cfg#140
[2]: https://chromium.googlesource.com/chromium/tools/build/+/3414fa041df5965be8a45b7cf7266045880ea202/masters/master.chromium.infra.cron/master.cfg#33
[3]: https://chrome-internal.googlesource.com/chrome/tools/build/+/bf490382b3aff4585b44e4a52629e49582d70e7e/masters/master.internal.client.v8/master.cfg#188

> This may not work for V8. The internal v8/trigger_proxy recipe may choose not to trigger other builders in case the isolate hash has not changed from last run

Wait, what is 'trigger_proxy' and how does it work?

> It does not seems to be supported in current or proposed design of the LUCI Scheduler (go/waterfall-scheduler-doc), hence it looks like a regression in LUCI.

go/waterfall-scheduler-doc is not a LUCI Scheduler design doc. No one is implementing this design (and it targets Buildbot). LUCI Scheduler design doc is go/luci-scheduler-v2 and it states "easy to write custom triggering policies" as one of the goals. It will be relatively straightforward to add "logarithmic algorithm", once v2 is completed.

> One would only need an API to peek at the current pending queue (should be available via buildbucket) and then decide whether to add new build request or replace an existing one with one that will combine the two. WDYT?

There's no really "pending queue" in buildbucket for Swarming builds. Buildbucket schedules tasks on Swarming immediately. "Replacing" a pending build involves canceling Swarming task and posting a new one. It is too hairy, IMHO.

> Wait, what is 'trigger_proxy' and how does it work?

This is a builder on internal.client.v8 master triggered by public builders on client.v8 via buckbucket, which in turn triggers other performance builders. The reason it exists, is that we don't want to expose internal builder names on a public master, but still want to trigger them.

> LUCI Scheduler design doc is go/luci-scheduler-v2
Thanks for sharing this link. I can now see the limitations of the current scheduler and why v2 version needs to be implemented. OTH, from the doc it's not clear how users would be able to write their own policies. Or do we plan to hard-code a few policies with some parameters and let users choose among them?

> There's no really "pending queue" in buildbucket for Swarming builds. Buildbucket schedules tasks on Swarming immediately. "Replacing" a pending build involves canceling Swarming task and posting a new one. It is too hairy, IMHO.

Oh, I see. I guess I was still thinking in the Buildbot model. Then perhaps recipes can talk to swarming directly and inspect its list of pending jobs for a given builder. Jobs would only be cancelled in case they have not started yet, otherwise they would not be on the pending list. One issue that I see with this is absence of transactionality: a pending job may have started by the time a recipe has decided that it wants to cancel and replace it. I am not sure if it's possible or safe to stop to terminate a running build on swarming.

IIUC, the same issue will affect Scheduler V2. On Buildbot, we had multiple build requests added to the pending queue and then processed together as soon as a slave has become available, so if 10 requests were scheduled while slave was busy, log_2(N) policy would cause 3 to be merged and scheduled as a single build. In LUCI Scheduler V2, we will have 10 invocations of a triggering job that add an entry to the list of triggers for a buildbucket job. Each addition will cause a triage to be run for the buildbucket job, which needs to run scheduling policy function. The latter is supposed to be a pure function, and hence can not check if bots are available on swarming, so it must decide whether to trigger a task or not without knowing how many triggers will be there by the time when we will have capacity to actually run a new build.

Therefore, it's still not clear to me how log-merge policy can be implemented in the LUCI Scheduler without being able to peek at the list of pending tasks in swarming. Or did I miss something in the design that makes it possible?

> Therefore, it's still not clear to me how log-merge policy can be implemented in the LUCI Scheduler without being able to peek at the list of pending tasks in swarming. Or did I miss something in the design that makes it possible?

The triggering policy knows the list of currently running invocations. In first iteration of this stuff we would be able to say something like:

job {
  triggering_policy {
    strategy: LOGARITHMIC_MERGE
    max_concurrent_invocations: 5
    ...
  }
}

The triage function will have access to a list of pending triggers (e.g. 10 of them), and list of currently running invocations (e.g. 2) and the config. It then can decide "I'm allowed to have 3 more invocations, so I can split 10 triggers between them as 5+3+2", and launch 3 more invocations.

In later iterations I'm considering teaching Scheduler to learn about available capacity dynamically and feed it to the policy function too. It would be a subject to race conditions though, as you've noticed, primarily affecting small pools. I'm still thinking about it... I don't believe this is strictly required in short term though (I think "max_concurrent_invocations" and carefully constructed pools may be sufficient as a start, dynamic capacity discovery is cost optimization).

Re "adding triggering policies". I plan to have an interface in Scheduler code and a bunch of implementations, identified by some enum. What implementation to use (and what parameters it gets) will be specified in the config. So if someone wants some exotic triggering policy, they'll have to write a chunk of Go code and ask us to redeploy scheduler service :( I'll try to make this as decoupled from Scheduler guts as possible, so it can be written and tested without groking the entirety of Scheduler codebase.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/d8ec75d36afb4290d44739384c622e19729fb3dd

commit d8ec75d36afb4290d44739384c622e19729fb3dd
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Thu Jan 25 23:42:41 2018

[scheduler] Rename InvocationRequest to task.Request and move to Controller.

Pure refactoring, no logic changes.

This is in preparation for adding move stuff (such as a build properties and
tags, derived from triggers) to the request.

R=tandrii@chromium.org, nodir@chromium.org
BUG= 794499 ,  764043 

Change-Id: I7bd35cc4b6cd631ce9ea31fda205fced880f5cce
Reviewed-on: https://chromium-review.googlesource.com/885507
Reviewed-by: Erik Staab <estaab@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/catalog/catalog_test.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/engine/engine_test.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/engine/engine_v2.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/engine/engine_v2_test.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/engine/invocation.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/engine/triage.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/engine/triage_test.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/buildbucket/buildbucket.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/buildbucket/buildbucket_test.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/gitiles/gitiles.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/gitiles/gitiles_test.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/noop/noop.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/noop/noop_test.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/task.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/urlfetch/urlfetch.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/urlfetch/urlfetch_test.go
[modify] https://crrev.com/d8ec75d36afb4290d44739384c622e19729fb3dd/scheduler/appengine/task/utils/tasktest/controller.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/2337be80bf794c93576604258f0c8d1e70eade4f

commit 2337be80bf794c93576604258f0c8d1e70eade4f
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Jan 26 00:01:01 2018

[scheduler] Move protos with trigger bodies from internal into API.

We'll add new BuildbucketTrigger proto there and then will use them in new
EmitTriggers RPC.

Keep the full-fledged Trigger (the one containing all metadata) in the internal
still, there's no need to expose it to the outside world for now.

Also drop "Data" suffix in the proto message names, it is redundant now.

R=tandrii@chromium.org, nodir@chromium.org
BUG= 794499 ,  764043 

Change-Id: Iba1508d71cacf3627cf43f01994466191d26dc85
Reviewed-on: https://chromium-review.googlesource.com/885623
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/api/scheduler/v1/pb.discovery.go
[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/api/scheduler/v1/scheduler.pb.go
[add] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/api/scheduler/v1/triggers.pb.go
[add] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/api/scheduler/v1/triggers.proto
[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/appengine/engine/engine_test.go
[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/appengine/internal/timers.pb.go
[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/appengine/internal/triggers.pb.go
[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/appengine/internal/triggers.proto
[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/appengine/task/buildbucket/buildbucket.go
[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/appengine/task/buildbucket/buildbucket_test.go
[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/appengine/task/gitiles/gitiles.go
[modify] https://crrev.com/2337be80bf794c93576604258f0c8d1e70eade4f/scheduler/appengine/task/gitiles/gitiles_test.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/b19575e236fd89932c9ded24f355efb6148636f1

commit b19575e236fd89932c9ded24f355efb6148636f1
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Jan 26 03:01:21 2018

[scheduler] Add EmitTriggers RPC and BuildbucketTrigger proto definitions.

Buildbucket triggers are not adequately processed yet.

R=tandrii@chromium.org
BUG= 794499 

Change-Id: Ibc840eaafddacdce50e2442a5d36d07ef73d2cd6
Reviewed-on: https://chromium-review.googlesource.com/887793
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/b19575e236fd89932c9ded24f355efb6148636f1/scheduler/api/scheduler/v1/pb.discovery.go
[modify] https://crrev.com/b19575e236fd89932c9ded24f355efb6148636f1/scheduler/api/scheduler/v1/scheduler.pb.go
[modify] https://crrev.com/b19575e236fd89932c9ded24f355efb6148636f1/scheduler/api/scheduler/v1/scheduler.proto
[modify] https://crrev.com/b19575e236fd89932c9ded24f355efb6148636f1/scheduler/api/scheduler/v1/triggers.pb.go
[modify] https://crrev.com/b19575e236fd89932c9ded24f355efb6148636f1/scheduler/api/scheduler/v1/triggers.proto
[modify] https://crrev.com/b19575e236fd89932c9ded24f355efb6148636f1/scheduler/appengine/apiservers/scheduler.go
[modify] https://crrev.com/b19575e236fd89932c9ded24f355efb6148636f1/scheduler/appengine/apiservers/scheduler_test.go
[modify] https://crrev.com/b19575e236fd89932c9ded24f355efb6148636f1/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/b19575e236fd89932c9ded24f355efb6148636f1/scheduler/appengine/internal/triggers.pb.go
[modify] https://crrev.com/b19575e236fd89932c9ded24f355efb6148636f1/scheduler/appengine/internal/triggers.proto

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/9647fda714506eeb471444c921b5ba7f6a7425c7

commit 9647fda714506eeb471444c921b5ba7f6a7425c7
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Fri Jan 26 19:57:37 2018

[scheduler] Allow to order triggers by time they were emitted.

This is needed to sort triggers (which are stored as unordered sets in v2
version of the engine) when there's no better comparator available.

Be careful to handle "emitted at exact same moment in time" situation correctly
by using tuple (timestamp, order in batch).

This CL populates the fields, but doesn't use them yet.

R=tandrii@chromium.org
BUG= 794499 ,  764043 

Change-Id: I5a25955eacddc10a2bf0a0b69ddb530eaeb2dda2
Reviewed-on: https://chromium-review.googlesource.com/888175
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/9647fda714506eeb471444c921b5ba7f6a7425c7/scheduler/api/scheduler/v1/pb.discovery.go
[modify] https://crrev.com/9647fda714506eeb471444c921b5ba7f6a7425c7/scheduler/api/scheduler/v1/scheduler.pb.go
[modify] https://crrev.com/9647fda714506eeb471444c921b5ba7f6a7425c7/scheduler/api/scheduler/v1/scheduler.proto
[modify] https://crrev.com/9647fda714506eeb471444c921b5ba7f6a7425c7/scheduler/appengine/apiservers/scheduler.go
[modify] https://crrev.com/9647fda714506eeb471444c921b5ba7f6a7425c7/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/9647fda714506eeb471444c921b5ba7f6a7425c7/scheduler/appengine/engine/engine_v2_test.go
[modify] https://crrev.com/9647fda714506eeb471444c921b5ba7f6a7425c7/scheduler/appengine/internal/triggers.pb.go
[modify] https://crrev.com/9647fda714506eeb471444c921b5ba7f6a7425c7/scheduler/appengine/internal/triggers.proto

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/60c57c108595388114eebe5eb911b59f32949364

commit 60c57c108595388114eebe5eb911b59f32949364
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Sat Jan 27 00:55:34 2018

[scheduler] Add a function that derives job properties from a set of triggers.

It doesn't do much yet. This is just a pure refactoring.

R=tandrii@chromium.org
BUG= 794499 

Change-Id: I244dc6e17bd479d0b7bbc539df8ffc4b3ccdc255
Reviewed-on: https://chromium-review.googlesource.com/890219
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/60c57c108595388114eebe5eb911b59f32949364/scheduler/appengine/engine/controller.go
[modify] https://crrev.com/60c57c108595388114eebe5eb911b59f32949364/scheduler/appengine/engine/engine.go
[add] https://crrev.com/60c57c108595388114eebe5eb911b59f32949364/scheduler/appengine/engine/request.go
[modify] https://crrev.com/60c57c108595388114eebe5eb911b59f32949364/scheduler/appengine/task/task.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/30571c938ab5407dbe7a455b915acf9eb88a50cd

commit 30571c938ab5407dbe7a455b915acf9eb88a50cd
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Sat Jan 27 03:47:14 2018

[scheduler] add TRIGGERER role.

TRIGGERER role provides ability to trigger a job.
Previously, OWNER role was required for this.

TRIGGERER role doesn't grant READER permission, allowing use case of
public job triggering internal jobs.

Also:
 * code has been simplified with less copy-pasta.
 * READER acl is no longer required for Job/Trigger definitions.

R=vadimsh@chromium.org

Bug:  794499 
Change-Id: I517542295fbcf8485d09f2f0d6cd11e287aa97b1
Reviewed-on: https://chromium-review.googlesource.com/888173
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/30571c938ab5407dbe7a455b915acf9eb88a50cd/scheduler/api/scheduler/v1/pb.discovery.go
[modify] https://crrev.com/30571c938ab5407dbe7a455b915acf9eb88a50cd/scheduler/api/scheduler/v1/scheduler.pb.go
[modify] https://crrev.com/30571c938ab5407dbe7a455b915acf9eb88a50cd/scheduler/api/scheduler/v1/scheduler.proto
[modify] https://crrev.com/30571c938ab5407dbe7a455b915acf9eb88a50cd/scheduler/appengine/acl/acl.go
[modify] https://crrev.com/30571c938ab5407dbe7a455b915acf9eb88a50cd/scheduler/appengine/acl/acl_test.go
[modify] https://crrev.com/30571c938ab5407dbe7a455b915acf9eb88a50cd/scheduler/appengine/apiservers/scheduler.go
[modify] https://crrev.com/30571c938ab5407dbe7a455b915acf9eb88a50cd/scheduler/appengine/apiservers/scheduler_test.go
[modify] https://crrev.com/30571c938ab5407dbe7a455b915acf9eb88a50cd/scheduler/appengine/catalog/catalog_test.go
[modify] https://crrev.com/30571c938ab5407dbe7a455b915acf9eb88a50cd/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/30571c938ab5407dbe7a455b915acf9eb88a50cd/scheduler/appengine/engine/engine_test.go
[modify] https://crrev.com/30571c938ab5407dbe7a455b915acf9eb88a50cd/scheduler/appengine/engine/job.go
[modify] https://crrev.com/30571c938ab5407dbe7a455b915acf9eb88a50cd/scheduler/appengine/messages/cron.pb.go
[modify] https://crrev.com/30571c938ab5407dbe7a455b915acf9eb88a50cd/scheduler/appengine/messages/cron.proto
[modify] https://crrev.com/30571c938ab5407dbe7a455b915acf9eb88a50cd/scheduler/appengine/ui/invocation.go
[modify] https://crrev.com/30571c938ab5407dbe7a455b915acf9eb88a50cd/scheduler/appengine/ui/job.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/e22ae915e2c148b71a0073601b370eb396c1570a

commit e22ae915e2c148b71a0073601b370eb396c1570a
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Sun Jan 28 22:40:02 2018

[scheduler] Generate invocation properties from pending triggers.

Most recent trigger always wins for now (until v2 engine is implemented).

This makes Buildbucket triggers work too. Build properties and tags supplied in
the trigger are passed to the buildbucket job.

Also adds some minimal UI for displaying properties and tags, to aid in
debugging.

R=tandrii@chromium.org
BUG= 794499 

Change-Id: I77af5164f7a7251519221fd0f03713ecd3c89327
Reviewed-on: https://chromium-review.googlesource.com/889922
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/e22ae915e2c148b71a0073601b370eb396c1570a/scheduler/appengine/devcfg/projects/project1/luci-scheduler-dev.cfg
[modify] https://crrev.com/e22ae915e2c148b71a0073601b370eb396c1570a/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/e22ae915e2c148b71a0073601b370eb396c1570a/scheduler/appengine/engine/invocation.go
[modify] https://crrev.com/e22ae915e2c148b71a0073601b370eb396c1570a/scheduler/appengine/engine/request.go
[modify] https://crrev.com/e22ae915e2c148b71a0073601b370eb396c1570a/scheduler/appengine/frontend/templates/pages/invocation.html
[modify] https://crrev.com/e22ae915e2c148b71a0073601b370eb396c1570a/scheduler/appengine/task/buildbucket/buildbucket.go
[modify] https://crrev.com/e22ae915e2c148b71a0073601b370eb396c1570a/scheduler/appengine/task/task.go
[modify] https://crrev.com/e22ae915e2c148b71a0073601b370eb396c1570a/scheduler/appengine/ui/presentation.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/2ddfc0997fb5d3fa29d530ab8bdda3b7e9156c44

commit 2ddfc0997fb5d3fa29d530ab8bdda3b7e9156c44
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Mon Jan 29 23:19:04 2018

[scheduler] Refactor how we check Read ACLs to reduce number of datastore ops.

Previously each individual engine method used to fetch job.Engine entity to
check read ACLs. This causes unnecessary duplicate reads, especially on UI
pages that call many engine methods to render singe page.

Instead read ACL is now checked only in GetVisibleJob (and similar), and other
methods accept *Job, assuming callers can get them only via GetVisibleJob (which
is true for public engine API).

The (possibly unfortunate) side effect of this is that EmitTriggers users now
require READER access. So TRIGGERER role now implies READER role. This may
change in the future.

Finally, extract some common code in helper methods (such as filterForRole for
batch ACL check in Engine, and jobFromEngine for getting *Job or HTTP 404 page
in UI).

R=tandrii@chromium.org
BUG= 794499 

Change-Id: I81bf9b33678b762a0932a13bf49e39c2187a9169
Reviewed-on: https://chromium-review.googlesource.com/890759
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/2ddfc0997fb5d3fa29d530ab8bdda3b7e9156c44/scheduler/appengine/acl/acl.go
[modify] https://crrev.com/2ddfc0997fb5d3fa29d530ab8bdda3b7e9156c44/scheduler/appengine/acl/acl_test.go
[modify] https://crrev.com/2ddfc0997fb5d3fa29d530ab8bdda3b7e9156c44/scheduler/appengine/apiservers/scheduler.go
[modify] https://crrev.com/2ddfc0997fb5d3fa29d530ab8bdda3b7e9156c44/scheduler/appengine/apiservers/scheduler_test.go
[modify] https://crrev.com/2ddfc0997fb5d3fa29d530ab8bdda3b7e9156c44/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/2ddfc0997fb5d3fa29d530ab8bdda3b7e9156c44/scheduler/appengine/engine/engine_test.go
[modify] https://crrev.com/2ddfc0997fb5d3fa29d530ab8bdda3b7e9156c44/scheduler/appengine/engine/engine_v2_test.go
[modify] https://crrev.com/2ddfc0997fb5d3fa29d530ab8bdda3b7e9156c44/scheduler/appengine/engine/job.go
[modify] https://crrev.com/2ddfc0997fb5d3fa29d530ab8bdda3b7e9156c44/scheduler/appengine/frontend/handler.go
[modify] https://crrev.com/2ddfc0997fb5d3fa29d530ab8bdda3b7e9156c44/scheduler/appengine/messages/cron.pb.go
[modify] https://crrev.com/2ddfc0997fb5d3fa29d530ab8bdda3b7e9156c44/scheduler/appengine/messages/cron.proto
[add] https://crrev.com/2ddfc0997fb5d3fa29d530ab8bdda3b7e9156c44/scheduler/appengine/ui/helpers.go
[modify] https://crrev.com/2ddfc0997fb5d3fa29d530ab8bdda3b7e9156c44/scheduler/appengine/ui/invocation.go
[modify] https://crrev.com/2ddfc0997fb5d3fa29d530ab8bdda3b7e9156c44/scheduler/appengine/ui/job.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/1264b193431e8f048c5dec8328689fb1a27e78fa

commit 1264b193431e8f048c5dec8328689fb1a27e78fa
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Mon Jan 29 23:30:41 2018

[scheduler] Display pending triggers in the UI.

Otherwise it is totally unclear whether EmitTriggers RPC has any effect at all.

R=tandrii@chromium.org
BUG= 794499 

Change-Id: I510e6384973d0e3f73a20deddb4aee9bb6897df7
Reviewed-on: https://chromium-review.googlesource.com/890860
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/1264b193431e8f048c5dec8328689fb1a27e78fa/scheduler/appengine/apiservers/scheduler_test.go
[modify] https://crrev.com/1264b193431e8f048c5dec8328689fb1a27e78fa/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/1264b193431e8f048c5dec8328689fb1a27e78fa/scheduler/appengine/engine/engine_v2_test.go
[modify] https://crrev.com/1264b193431e8f048c5dec8328689fb1a27e78fa/scheduler/appengine/engine/job.go
[modify] https://crrev.com/1264b193431e8f048c5dec8328689fb1a27e78fa/scheduler/appengine/engine/request.go
[modify] https://crrev.com/1264b193431e8f048c5dec8328689fb1a27e78fa/scheduler/appengine/engine/triage.go
[modify] https://crrev.com/1264b193431e8f048c5dec8328689fb1a27e78fa/scheduler/appengine/engine/utils.go
[modify] https://crrev.com/1264b193431e8f048c5dec8328689fb1a27e78fa/scheduler/appengine/frontend/templates/pages/job.html
[modify] https://crrev.com/1264b193431e8f048c5dec8328689fb1a27e78fa/scheduler/appengine/ui/job.go
[modify] https://crrev.com/1264b193431e8f048c5dec8328689fb1a27e78fa/scheduler/appengine/ui/presentation.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/c3c1b368036e5237850adf586a363a05fee9c878

commit c3c1b368036e5237850adf586a363a05fee9c878
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Mon Jan 29 23:38:53 2018

[scheduler] Keep track of who is submitting triggers.

Populate TriggeredBy invocation field based on the trigger that provoked the
invocation. Display emails of triggerrers in trigger lists in UI.

R=tandrii@chromium.org
BUG= 794499 

Change-Id: Ic2fa8e548dc4fae9f5d1642896231f8f9cb282de
Reviewed-on: https://chromium-review.googlesource.com/890701
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/c3c1b368036e5237850adf586a363a05fee9c878/scheduler/appengine/apiservers/scheduler.go
[modify] https://crrev.com/c3c1b368036e5237850adf586a363a05fee9c878/scheduler/appengine/engine/engine.go
[modify] https://crrev.com/c3c1b368036e5237850adf586a363a05fee9c878/scheduler/appengine/engine/engine_v2_test.go
[modify] https://crrev.com/c3c1b368036e5237850adf586a363a05fee9c878/scheduler/appengine/engine/request.go
[modify] https://crrev.com/c3c1b368036e5237850adf586a363a05fee9c878/scheduler/appengine/frontend/templates/includes/base.html
[modify] https://crrev.com/c3c1b368036e5237850adf586a363a05fee9c878/scheduler/appengine/internal/triggers.pb.go
[modify] https://crrev.com/c3c1b368036e5237850adf586a363a05fee9c878/scheduler/appengine/internal/triggers.proto
[modify] https://crrev.com/c3c1b368036e5237850adf586a363a05fee9c878/scheduler/appengine/ui/presentation.go

The server side is ready and has been deployed to luci-scheduler-dev.appspot.com and luci-scheduler.appspot.com.

As mentioned several times already, the implementation based on v1 scheduler supports only one task running concurrently, and when it starts it consumes all pending triggers (aka build requests in Buildbot terminology). It picks properties for the build from the most recent trigger, totally ignoring all the rest. I believe this should be sufficient for Chromium builder/tester splits. v2 schedule will make this more flexible.

Next steps are:
1. Add 'prpc' tool to PATH in LUCI build environment.
2. Write 'scheduler' recipe module that can emit triggers via EmitTriggers RPC using 'prpc' tool.
3. Add new necessary luci-scheduler.cfg config entries for testers.
4. Modify chromium recipes to emit triggers via 'scheduler' recipe module instead of Buildbucket.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/luci-go.git/+/7157d89e72468abc2183b371a9e290b4e622c481

commit 7157d89e72468abc2183b371a9e290b4e622c481
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Tue Jan 30 00:33:34 2018

[prpc] Rename "rpc" binary to "prpc".

This prepares this binary for use in recipes, such as upcoming
luci-scheduler recipe module.

R=nodir, vadimsh

Bug:  794499 
Change-Id: I8f5c2d57d7d8ae19796452cb32f883e76658088a
Reviewed-on: https://chromium-review.googlesource.com/891838
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Nodir Turakulov <nodir@chromium.org>

[rename] https://crrev.com/7157d89e72468abc2183b371a9e290b4e622c481/grpc/cmd/prpc/call.go
[rename] https://crrev.com/7157d89e72468abc2183b371a9e290b4e622c481/grpc/cmd/prpc/descriptor.go
[rename] https://crrev.com/7157d89e72468abc2183b371a9e290b4e622c481/grpc/cmd/prpc/doc.go
[rename] https://crrev.com/7157d89e72468abc2183b371a9e290b4e622c481/grpc/cmd/prpc/f2j.go
[rename] https://crrev.com/7157d89e72468abc2183b371a9e290b4e622c481/grpc/cmd/prpc/fmt.go
[rename] https://crrev.com/7157d89e72468abc2183b371a9e290b4e622c481/grpc/cmd/prpc/format.go
[rename] https://crrev.com/7157d89e72468abc2183b371a9e290b4e622c481/grpc/cmd/prpc/generate.go
[rename] https://crrev.com/7157d89e72468abc2183b371a9e290b4e622c481/grpc/cmd/prpc/j2f.go
[rename] https://crrev.com/7157d89e72468abc2183b371a9e290b4e622c481/grpc/cmd/prpc/j2f_test.go
[rename] https://crrev.com/7157d89e72468abc2183b371a9e290b4e622c481/grpc/cmd/prpc/main.go
[rename] https://crrev.com/7157d89e72468abc2183b371a9e290b4e622c481/grpc/cmd/prpc/printer.go
[rename] https://crrev.com/7157d89e72468abc2183b371a9e290b4e622c481/grpc/cmd/prpc/printer_test.desc
[rename] https://crrev.com/7157d89e72468abc2183b371a9e290b4e622c481/grpc/cmd/prpc/printer_test.go
[rename] https://crrev.com/7157d89e72468abc2183b371a9e290b4e622c481/grpc/cmd/prpc/printer_test.pb_test.go
[rename] https://crrev.com/7157d89e72468abc2183b371a9e290b4e622c481/grpc/cmd/prpc/printer_test.proto
[rename] https://crrev.com/7157d89e72468abc2183b371a9e290b4e622c481/grpc/cmd/prpc/show.go

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/4584a6323a8261c535accbd7208fdd52bc9f4f2f

commit 4584a6323a8261c535accbd7208fdd52bc9f4f2f
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Jan 30 00:54:08 2018

Roll luci-go DEPS.

infra/go/src/go.chromium.org/luci:
7157d89e7 [prpc] Rename "rpc" binary to "prpc".
c3c1b3680 [scheduler] Keep track of who is submitting triggers.
1264b1934 [scheduler] Display pending triggers in the UI.
2ddfc0997 [scheduler] Refactor how we check Read ACLs to reduce number of datastore ops.
4ed7d90b6 [Machine Database] Add command to delete hosts
d3692e446 [milo] Set min active instances
0257f2784 [Machine Database] Allow VMs to have different VLANs than their hosts
007c389df [Machine Database] Add state to switches
35b6729b1 [milo] fix error message format
e22ae915e [scheduler] Generate invocation properties from pending triggers.
53d25bb98 Fix link to spec.proto.
30571c938 [scheduler] add TRIGGERER role.
4a773a8a2 [Machine Database] Add commands to create, retrieve physical hosts
885521fdc [Machine Database] Add command to delete machines
ed81eb783 Add int32 flag type
ad59ef085 [Machine Database] Add state to racks
50a31ebef [Machine Database] Support adding, retrieving vms
60c57c108 [scheduler] Add a function that derives job properties from a set of triggers.

R=tandrii@chromium.org
BUG= 794499 

Change-Id: Id9f2c2e377f178f070a2622a4d9bdf1528cef074
Reviewed-on: https://chromium-review.googlesource.com/892298
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/4584a6323a8261c535accbd7208fdd52bc9f4f2f/DEPS

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/702dcaa7cfcb5367873c63ad969fb106e4f1d0ab

commit 702dcaa7cfcb5367873c63ad969fb106e4f1d0ab
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Jan 30 01:09:06 2018

Add cipd package with prpc binary.

To be deployed in LUCI build environment.

R=tandrii@chromium.org
BUG= 794499 

Change-Id: I98d2757c9212415e50056adacb6739f06688e163
Reviewed-on: https://chromium-review.googlesource.com/892057
Commit-Queue: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>

[add] https://crrev.com/702dcaa7cfcb5367873c63ad969fb106e4f1d0ab/build/packages/prpc.yaml

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/17ecbb1825b41bdb67312a50d62821ca83431c10

commit 17ecbb1825b41bdb67312a50d62821ca83431c10
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Jan 30 02:30:02 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/17c55dcc7f402f13ad593c350bb95111a4bf6cd7

commit 17c55dcc7f402f13ad593c350bb95111a4bf6cd7
Author: Vadim Shtayura <vadimsh@chromium.org>
Date: Tue Jan 30 20:17:19 2018

Andrii is adding triggering support to recipes.

recipe https://chromium-review.googlesource.com/c/infra/luci/recipes-py/+/892699

now hacking end-to-end test + chromium recipe change.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/b438ab6420331d127ae321c45bfbc6435487fdfc

commit b438ab6420331d127ae321c45bfbc6435487fdfc
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu Feb 01 22:37:13 2018

Define Linux Tests SANDBOX for triggering through scheduler.

Also, rename and temporarily not gitiles-trigger corresponding Builder
(compiler).

TBR=hinoka@chromium.org

Bug:  794499 
Change-Id: I3941efd7b25583ccc0d88c400d92ff62d683f8b9
Reviewed-on: https://chromium-review.googlesource.com/897469
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/b438ab6420331d127ae321c45bfbc6435487fdfc/luci-scheduler.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/41f0b16b122db63a209bdcd2016633eb1fdbf2e2

commit 41f0b16b122db63a209bdcd2016633eb1fdbf2e2
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Thu Feb 01 22:46:59 2018

scheduler: experiment with TRIGGERER permission on SANDBOX.

R=vadimsh

Bug:  794499 
Change-Id: I0a10f8802862d20a3171c8267b95606192588d48
Reviewed-on: https://chromium-review.googlesource.com/898343
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/41f0b16b122db63a209bdcd2016633eb1fdbf2e2/luci-scheduler.cfg

Resolved lack of led-access, now resolving bugs in recipe module and in chromium recipe integration.

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/recipes-py/+/c9374582f3cbe9b1e54eaa851b891de56b7b7aa9

commit c9374582f3cbe9b1e54eaa851b891de56b7b7aa9
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Fri Feb 02 23:00:39 2018

Add scheduler recipe module with EmitTrigger RPC support only.

R=vadimsh@chromium.org

Bug:  794499 
Change-Id: Ie5057e7bbf87dbc3304635c8b2b9627780086165
Reviewed-on: https://chromium-review.googlesource.com/892699
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Robbie Iannucci <iannucci@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[add] https://crrev.com/c9374582f3cbe9b1e54eaa851b891de56b7b7aa9/recipe_modules/scheduler/examples/emit_triggers.py
[add] https://crrev.com/c9374582f3cbe9b1e54eaa851b891de56b7b7aa9/recipe_modules/scheduler/__init__.py
[add] https://crrev.com/c9374582f3cbe9b1e54eaa851b891de56b7b7aa9/recipe_modules/scheduler/OWNERS
[modify] https://crrev.com/c9374582f3cbe9b1e54eaa851b891de56b7b7aa9/README.recipes.md
[modify] https://crrev.com/c9374582f3cbe9b1e54eaa851b891de56b7b7aa9/recipe_modules/buildbucket/api.py
[add] https://crrev.com/c9374582f3cbe9b1e54eaa851b891de56b7b7aa9/recipe_modules/scheduler/api.py
[add] https://crrev.com/c9374582f3cbe9b1e54eaa851b891de56b7b7aa9/recipe_modules/scheduler/examples/emit_triggers.expected/basic.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/57ba78020d9bdb21a25e2cf4a16ffdca83ce4a22

commit 57ba78020d9bdb21a25e2cf4a16ffdca83ce4a22
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Fri Feb 02 23:47:48 2018

chromium CI tests: trigger testers through luci-scheduler.

Previously, it was done through buildbucket which doesn't support
build merging, while luci-scheduler does. This therefore replicates
behavior of buildbot which did build merging.

Tested end-to-end with led:
  * builder build
    https://ci.chromium.org/swarming/task/3b6cd823135f0910?server=chromium-swarm.appspot.com
  * triggered tester on scheduler
    https://luci-scheduler.appspot.com/jobs/chromium/Linux%20Tests%20SANDBOX/9121173095042174960

Depends on:
  https://chromium-review.googlesource.com/c/infra/luci/recipes-py/+/892699

Bug:  794499 
Change-Id: If39c1e8f927277522edfadb69e5235e6d37135fa
Reviewed-on: https://chromium-review.googlesource.com/898505
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>

[modify] https://crrev.com/57ba78020d9bdb21a25e2cf4a16ffdca83ce4a22/scripts/slave/README.recipes.md
[modify] https://crrev.com/57ba78020d9bdb21a25e2cf4a16ffdca83ce4a22/scripts/slave/recipe_modules/chromium_tests/api.py
[modify] https://crrev.com/57ba78020d9bdb21a25e2cf4a16ffdca83ce4a22/scripts/slave/recipe_modules/chromium_tests/__init__.py
[modify] https://crrev.com/57ba78020d9bdb21a25e2cf4a16ffdca83ce4a22/scripts/slave/recipe_modules/chromium_tests/tests/api/main_waterfall_steps.expected/builder.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/c4aaa23a80d63715986b7832f1d90abb76816933

commit c4aaa23a80d63715986b7832f1d90abb76816933
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Sat Feb 03 01:02:51 2018

[luci-scheduler.cfg] bring back gitiles-triggering of sandbox builder.

TBR=vadimsh

Bug:  794499 
Change-Id: I9c80646033bb8782a9cd9f8d09e4f70bc75001e8
Reviewed-on: https://chromium-review.googlesource.com/900316
Reviewed-by: Andrii Shyshkalov <tandrii@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/c4aaa23a80d63715986b7832f1d90abb76816933/luci-scheduler.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/87c849c6257c7594ed558839a2c501a0d4f08437

commit 87c849c6257c7594ed558839a2c501a0d4f08437
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Sat Feb 03 01:48:58 2018

[luci-scheduler.cfg] prepare for addition of builder-Testers.

These are builds triggered not by new commits in gitiles, but
by other builders, whcih, confusingly, are also called "Builder"s.

R=vadimsh

Bug:  794499 
Change-Id: I732bf13c0ba0f0df71ee03bd6faf00fe1d6ae57d
Reviewed-on: https://chromium-review.googlesource.com/900293
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/87c849c6257c7594ed558839a2c501a0d4f08437/luci-scheduler.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ff01e09dc891da212d65db17b85c39317f099a00

commit ff01e09dc891da212d65db17b85c39317f099a00
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Sat Feb 03 02:48:27 2018

Add "Marshmallow 64 bit Tester" child-tester to luci-scheduler.cfg

It is triggered by "Android arm64 Builder (dbg)" builder.

R=bpastene@chromium.org

No-Tree-Checks: True
Bug:  794499 
Change-Id: Ia17870fe273bd55f71a1e27fcc15f4b8fe9a68ba
Reviewed-on: https://chromium-review.googlesource.com/900490
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Benjamin Pastene <bpastene@chromium.org>

[modify] https://crrev.com/ff01e09dc891da212d65db17b85c39317f099a00/luci-scheduler.cfg

What has been done:
 * triggering through scheduler works: https://ci.chromium.org/p/chromium/g/sandbox-side-by-side/console 
 * build merging on scheduler works too. For example,
     https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Tests%20SANDBOX/4884
   has actually merged two triggers, see
     https://luci-scheduler.appspot.com/jobs/chromium/Linux%20Tests%20SANDBOX/9121077535104961632

What remains here:
1. add configs for all builder-testers pairs that are being migrated (all CLs done, landing...)
2. update go/migrate2luci doc.
3. send heads up to task force mailing list.
4. cleanup:
4.1. issue 808677 to deploy prpc with depot_tools
4.2. use classes in scheduler recipe.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/f9270680d4ccbc89d78ed7b735f2d7adf65d9bde

commit f9270680d4ccbc89d78ed7b735f2d7adf65d9bde
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Sat Feb 03 02:55:08 2018

Add "Linux Tests" and "Linux Tests (dbg)(1)" to luci-scheduler.cfg.

Also, re-order their presence in migration console.

R=hinoka, martiniss, vadimsh

No-Tree-Checks: True
Bug:  794499 
Change-Id: Icc2f70382d0c0b1444083660f0a1195aeb4469d1
Reviewed-on: https://chromium-review.googlesource.com/900492
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/f9270680d4ccbc89d78ed7b735f2d7adf65d9bde/luci-milo.cfg
[modify] https://crrev.com/f9270680d4ccbc89d78ed7b735f2d7adf65d9bde/luci-scheduler.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fc7b28d30d451d0b231d0f09cae0178441d41537

commit fc7b28d30d451d0b231d0f09cae0178441d41537
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Sat Feb 03 03:04:25 2018

Add "Mac10.9 Tests (dbg)" builder to luci-scheduler.cfg.

R=jbudorick

No-Tree-Checks: True
Bug:  794499 
Change-Id: Ia615bdc8d15499a9ced27e7264df0e7136189f51
Reviewed-on: https://chromium-review.googlesource.com/900493
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/fc7b28d30d451d0b231d0f09cae0178441d41537/luci-scheduler.cfg

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6b6d79d8dbac476660c2645e89a7f89a34f41466

commit 6b6d79d8dbac476660c2645e89a7f89a34f41466
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Mon Feb 05 23:49:57 2018

Luci-scheduler: define noop jobs to get triggering working.

This will get triggering working, but will no have side effects.
Once triggered builders are defined in buildbucket, the noop jobs
should be replaced with buildbucket jobs already defined in place, but
for now commented out.

R=bpastene, martiniss, vadimsh

Bug:  794499 
Change-Id: I9a9d51cc0a284bf08103483304fa9bd5e15344ae
Reviewed-on: https://chromium-review.googlesource.com/903048
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/6b6d79d8dbac476660c2645e89a7f89a34f41466/luci-scheduler.cfg

> 1. add configs for all builder-testers pairs that are being migrated (all CLs done, landing...)

Done.

> 2. update go/migrate2luci doc.

Done.

> 3. send heads up to task force mailing list.

Done: https://groups.google.com/a/google.com/d/msg/luci-task-force/iCgzqJqZJaI/SFCkK0HbAgAJ 

> 4. cleanup:
> 4.1. issue 808677 to deploy prpc with depot_tools
  in progress.

> 4.2. use classes in scheduler recipe.
CL https://chromium-review.googlesource.com/c/infra/luci/recipes-py/+/903264

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/recipes-py/+/3bc7426647d70d8ba58b895becc276ac3518917c

commit 3bc7426647d70d8ba58b895becc276ac3518917c
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Wed Feb 07 02:41:04 2018

scheduler recipe module: use classes instead of raw dicts.

Also fixes a bug where tags with value None weren't dropped,
as was intended.

TBR=iannucci@chromium.org

Bug:  794499 
Change-Id: I3642bc5c81b6dd98a03462a5b31183409036fbe6
Reviewed-on: https://chromium-review.googlesource.com/903264
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>

[modify] https://crrev.com/3bc7426647d70d8ba58b895becc276ac3518917c/recipe_modules/scheduler/api.py
[modify] https://crrev.com/3bc7426647d70d8ba58b895becc276ac3518917c/README.recipes.md
[modify] https://crrev.com/3bc7426647d70d8ba58b895becc276ac3518917c/recipe_modules/scheduler/examples/emit_triggers.py
[modify] https://crrev.com/3bc7426647d70d8ba58b895becc276ac3518917c/recipe_modules/scheduler/examples/emit_triggers.expected/basic.json

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/ed17c2a179d2e767c159b36a9664ff3091e4cf24

commit ed17c2a179d2e767c159b36a9664ff3091e4cf24
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Wed Feb 07 21:46:57 2018

remove no longer used recipe_module DEPS.

R=martiniss@chromium.org, vadimsh@chromium.org

Bug:  794499 
Change-Id: I03380d934dc616928664aeb891e8af8185506cb9
Reviewed-on: https://chromium-review.googlesource.com/899856
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/ed17c2a179d2e767c159b36a9664ff3091e4cf24/scripts/slave/README.recipes.md
[modify] https://crrev.com/ed17c2a179d2e767c159b36a9664ff3091e4cf24/scripts/slave/recipe_modules/chromium_tests/__init__.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/tools/build/+/919785eb7e8361a22a107d5e46e129a2f1803bba

commit 919785eb7e8361a22a107d5e46e129a2f1803bba
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Wed Feb 07 21:52:18 2018

chromium_tests: upgrade to new scheduler recipe_module api.

R=vadimsh

Bug:  794499 
Change-Id: Id9594491e9b864722dbf2c4774bfb2dfc12ac0ad
Reviewed-on: https://chromium-review.googlesource.com/905994
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Reviewed-by: Benjamin Pastene <bpastene@chromium.org>
Reviewed-by: Stephen Martinis <martiniss@chromium.org>
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>

[modify] https://crrev.com/919785eb7e8361a22a107d5e46e129a2f1803bba/scripts/slave/recipe_modules/chromium_tests/api.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/luci/recipes-py/+/baa8aadd3db91c4870bcc82c1091f25944dfdc96

commit baa8aadd3db91c4870bcc82c1091f25944dfdc96
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Wed Feb 07 22:41:05 2018

scheduler recipe_module: remove deprecated method.

R=estaab@chromium.org, vadimsh@chromium.org

Bug:  794499 
Change-Id: I5bb6e801aeff90f9237fa994402fdc3f0a322136
Reviewed-on: https://chromium-review.googlesource.com/905995
Commit-Queue: Andrii Shyshkalov <tandrii@chromium.org>
Reviewed-by: Erik Staab <estaab@chromium.org>

[modify] https://crrev.com/baa8aadd3db91c4870bcc82c1091f25944dfdc96/recipe_modules/scheduler/api.py
[modify] https://crrev.com/baa8aadd3db91c4870bcc82c1091f25944dfdc96/README.recipes.md
[modify] https://crrev.com/baa8aadd3db91c4870bcc82c1091f25944dfdc96/recipe_modules/scheduler/examples/emit_triggers.py

This has been fully completed.

Andrii, can you please shortly explain how does one trigger a build from a recipe via scheduler? Or just point me to some doc about it.

https://chromium-review.googlesource.com/898505 is how i converted from buildbucket to scheduler triggering. Note, that I slightly changed API of scheduler so see this as well https://chromium-review.googlesource.com/905994