There's state "lease_expiration_ts" when a bot is controlled by Machine Provider but this is not clear for troopers.

I'm not sure how to accomplish this within the existing Swarming datastore model. Are we okay with making Swarming have deep knowledge of the Machine Provider service? Right now the integration is tacked on, but it sounds like we will need the actual BotInfo to contain Machine Provider-related information if we want to do all this.

I don't think Swarming needs a deep knowledge of MP. Like how Swarming knows that Isolate exists and can store references to input and output digests, and create links to the service on the Web UI, it doesn't really understand much about it.

So my idea is really just to create links to the MP's bot page when a bot is managed by MP.

Right, and how do we know when it's an MP bot? If lease_expiration_ts in state then show links? But we don't know the bot's lease ID or anything unless it's stored in BotInfo.

components.machine_provider does.

I mean there's a need for feedback mechanism in a way or another for auto-scaling? Swarming needs to know what can be scaled and what can't.

To be clear, my proposal is better defined in the email thread we had on August 19th. I don't want to give Swarming any additional knowledge about MP than it already has. I just would like to move the knowledge it already has into BotInfo. That way all MP-related information is available when looking at the bot entities. Right now Swarming bot info is in BotInfo and the Machine Provider bot info is in MachineType and it's hard to correlate this information.

To quote the email:
"Would it be possible to put placeholder BotInfo entities in the datastore representing bots we want to connect from MP? We could have a job which scans these and requests MP bots matching the dimensions that are specified, and when the bot comes online it will self-report its finer-grain dimensions, updating the entity.

This way the same entity can keep track of the Swarming bot and its MP information like when the lease expires. I think that using the same entity will be necessary in order for /restricted/bots to easily display that it's an MP bot, and maybe display some controls like you had wanted."

In this proposal, to request an MP VM create a BotInfo (key=client request ID, this is an already existing unique ID used to deduplicate MP requests) with the desired Swarming dimensions and some flag like MP=true. Cron job sweeps BotInfos with MP=true and requests MP VM based on desired dimensions and updates BotInfo with lease ID of the request. When lease is fulfilled, update BotInfo with lease_expiration_ts. When VM connects to Swarming, make sure its ID is the client request ID (since bot ID is not required to be the same as hostname). It will report all of its usual Swarming dimensions and the BotInfo will be updated.

When rendering /restricted/bots UI, if MP=true and lease ID is not None then show a link to machine-provider.appspot.com/leases/<lease ID>.

All of this info is currently known by Swarming already, but hierarchically in the datastore it's not structured in a way we can easily get MP info for a bot when talking about a Swarming bot that came from MP.

If this proposal is unacceptable, please provide alternatives. Otherwise I'll start a doc about this proposal.

What do you mean? components.machine_provider doesn't know anything, it's just the API to issue requests to MP.

I will have to dig into the current behavior of BotInfo in more detail, but at a high-level this sounds like it will work.

Are you suggesting that MachineType is still required in order to specify the quantity and dimensions required from MP? So basically BotInfo entities would be created to match the desired quantities, and then another task would make the appropriate requests to lease bots from these entities?

I spoke to maruel over IM on Friday and this plan seems okay.

Yes, it's how you're suggesting. MachineType is still required to specify the kinds of machines needed from MP, but rather than storing a list of lease requests in the MT entity, we will create BotInfos.

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/4ad6fd1e59b4095a6876974a93b30d1bec5c9649

commit 4ad6fd1e59b4095a6876974a93b30d1bec5c9649
Author: smut <smut@google.com>
Date: Mon Oct 03 23:29:12 2016

Reimplement Machine Provider integration

BUG= 627560 

Review-Url: https://codereview.chromium.org/2386793002

[modify] https://crrev.com/4ad6fd1e59b4095a6876974a93b30d1bec5c9649/appengine/components/components/machine_provider/utils.py
[modify] https://crrev.com/4ad6fd1e59b4095a6876974a93b30d1bec5c9649/appengine/swarming/cron.yaml
[modify] https://crrev.com/4ad6fd1e59b4095a6876974a93b30d1bec5c9649/appengine/swarming/handlers_backend.py
[modify] https://crrev.com/4ad6fd1e59b4095a6876974a93b30d1bec5c9649/appengine/swarming/handlers_test.py
[modify] https://crrev.com/4ad6fd1e59b4095a6876974a93b30d1bec5c9649/appengine/swarming/queue.yaml
[modify] https://crrev.com/4ad6fd1e59b4095a6876974a93b30d1bec5c9649/appengine/swarming/server/lease_management.py
[modify] https://crrev.com/4ad6fd1e59b4095a6876974a93b30d1bec5c9649/appengine/swarming/server/lease_management_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/ef58021b34ac471545c8d7e9213246de045f2640

commit ef58021b34ac471545c8d7e9213246de045f2640
Author: smut <smut@google.com>
Date: Fri Oct 07 22:14:36 2016

Create a BotEvent when a bot is leased from Machine Provider

Creating a BotEvent also ensures a BotInfo exists.

BUG= 627560 

Review-Url: https://codereview.chromium.org/2402543002

[modify] https://crrev.com/ef58021b34ac471545c8d7e9213246de045f2640/appengine/swarming/handlers_bot_test.py
[modify] https://crrev.com/ef58021b34ac471545c8d7e9213246de045f2640/appengine/swarming/server/bot_management.py
[modify] https://crrev.com/ef58021b34ac471545c8d7e9213246de045f2640/appengine/swarming/server/bot_management_test.py
[modify] https://crrev.com/ef58021b34ac471545c8d7e9213246de045f2640/appengine/swarming/server/lease_management.py
[modify] https://crrev.com/ef58021b34ac471545c8d7e9213246de045f2640/appengine/swarming/swarming_rpcs.py

The following revision refers to this bug:
  https://chromium.googlesource.com/external/github.com/luci/luci-py.git/+/8b08401c7246b861acc2636d5e4edc4aa5694b33

commit 8b08401c7246b861acc2636d5e4edc4aa5694b33
Author: smut <smut@google.com>
Date: Wed Oct 12 17:38:11 2016

Don't assume existence of dimensions when querying for Swarming bots

BUG= 627560 

Review-Url: https://codereview.chromium.org/2412513003

[modify] https://crrev.com/8b08401c7246b861acc2636d5e4edc4aa5694b33/client/swarming.py