Also +Kevin for any tips/suggestions as Skia seems to be the only pool with chromeos devices in swarming at the moment:
https://chromium-swarm.appspot.com/botlist?c=id&c=os&c=task&c=status&f=os%3AChromeOS&l=100&q=os%3A&s=id%3Aasc

Each ChromeOS device has a host associated with it (e.g. swarming is NOT running on the ChromeOS device)

We have our ChromeOS devices on the same wired network as their swarming hosts (a Raspberry Pi).  
Each swarming host has a text file with an ip address of the ChromeOS device.
skia_mobile.py (bot_config) has logic to look for the file and then connect to the ChromeOS device via ssh.

Our recipes know how to read the text file and communicate with ChromeOS device to run tests.

I did not attempt to get swarming to run on the ChromeOS device because it was ... daunting.

I did make the bot work on crouton a few years back but there's several issues to this, so it's probably safer maintenance wise to use the dedicated host approach. I'm a fan of odroid-c1, beaglebones and RPis, e.g. https://github.com/periph/bootstrap

 - After talking with johnw, I discovered that 3 dell r230s were allocated along with the 30 chromebooks. If we were to go with the external controller approach, that would be 10 swarming bots per server. Not sure if each bot would need to be isolated from each other.

 - These devies have 6 cpus, 4 gb mem, ~20 gb disks. We've got bots with similar hardware specs in swarming today:
low end win laptop: https://chromium-swarm.appspot.com/bot?id=build150-b1
macbook air: https://chromium-swarm.appspot.com/bot?id=build123-b1

- So running on the swarming bot on the device might not be totally unrealistic. Though that's a pretty small disk... A randomly-picked chrome-perf swarming isolate is 9 GB. That's cutting it awfully close.

- Still no clue just how reliable the devices are. The 2 devices that were fully provisioned are still online after four months of idling. Though, operating under test load may change that story. If only I could get my local device flashed...

The low spec win laptop has an HDD, the Chromebooks have flash disk, that makes a huge difference in performance.

The size of an isolated cache doesn't matter that much, it gets filed with old executables (churn rate) until they are lazily evicted. The old executables will be evicted sooner.

It's worth nothing that the local bot cache doesn't evict files not touched since N days. Filed  issue 801870 .

I noticed that these chromebooks also have a slot for a microsd card. Depending on how the os handles mounting a slotted sd card at boot, putting one in might negate any affects of limited disk space. I'll see how far I can get a bot running on the device itself.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/511f47a19d368bfd68defc214d8142647a387be7

commit 511f47a19d368bfd68defc214d8142647a387be7
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Wed Jan 24 02:46:16 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/780015cfaeb8dc2ed6d2db2feb71d415c1e6cfe3

commit 780015cfaeb8dc2ed6d2db2feb71d415c1e6cfe3
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Wed Jan 24 03:14:28 2018

Amazingly, I got one of the bots into swarming with very little finagling:
https://chromium-swarm-dev.appspot.com/bot?id=build2-a1

It's stable, but complaining about a lack of various auth/token files that are expected on most of our swarming bots. These are deployed via puppet, which of course isn't running on the chromebook.

Though somehow I managed to install ruby (don't ask how), and subsequently installed puppet via gem. So now the chromebook also has a puppet binary. Running the agent of course complains about cert/master errors. I'll need to work with labs to see if we can get that setup, but for the moment it seems like we'll be able to run both the swarming bot and a puppet agent on the devices themselves. (Who needs docker?!)

Side notes:
- "arm64-32" looks like arm64 CPU and 32 bits user mode? Is that right?
- Thermal sensors even work!
- I didn't recall I had used 'Chromeos' for the OS. Maybe worth fixing the case while it won't do too much damage? 'Chromeos-' is very useless.

You'll have to document this ruby on chromeos, otherwise you are lowering the bus factor. :) Congrats!

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/20606f008d670ee2af0e8f4bb699c367060bc494

commit 20606f008d670ee2af0e8f4bb699c367060bc494
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Wed Jan 24 19:27:05 2018

Hmmm, puppet may still be a no-go. Running the binary exits early with an error about /etc/ being read-only. It seems that puppet refuses to use anything other than /etc/puppetlabs as its config dir, and can't be pointed elsewhere.

Trying to remount / as rw exits with "cannot remount /dev/dm-0 read-write, is write-protected". I found some pointers to a bash script on the device that can supposedly remount the rootfs, but it doesn't seem to be working. I'll keep poking around, there's gotta be some way to get this running...

And I fixed up the dimensions a bit. I'm impressed that the swarming bot knew it was cros right out of the box, but I added some hacky overwrites in https://chrome-internal.googlesource.com/infradata/config/+/master/configs/chromium-swarm-dev/scripts/cros.py 

Making puppet work on ChromeOS is probably very hard and it will be breaking all the time :( Most of our manifests probably won't work with ChromeOS anyway.

What exactly do we want from "puppet on ChromeOS"? Maybe there's a simpler alternative.

Yeah, "puppet on ChromeOS" is almost certainly the wrong solution. 

I think we  need to make sure whatever we need to run on the device is either built and included in the dev CrOS image that is getting flashed onto the device, or pushed down to the device at test runtime, just like we do on Android.

I disagree that it's "certainly the wrong solution". Having spent more time than anyone else here on problems with having the device and controller being separate entities (android), I believe it would be much more ideal if we could contain everything on a single machine. And, judging by my progress so far (and how little effort was needed to get there), I still think it's a reasonable option. That said, if things become too hairy and I can't iron out the kinks/hacks, I'll start exploring some funky ubuntu-container ssh hookup.

And to answer your question Vadim, I'm thinking we'd just need service account files, netrc files, git cookie files, boto files, and pretty much any other auth-related creds that puppet normally deploys. We could skip package/service installations for now.

For LUCI builders we actually don't need service account files, netrc files, git cookie files, boto files, ... All we need is a machine certificate (+ private key) to authenticate to Swarming, and the rest of auth stuff follows from there (this is the whole point on go/swarming-service-accounts - get rid of the zoo of predeployed static secrets).

For getting the certificate I can adapt the solution we use in GCE (https://chrome-internal.googlesource.com/infra/infra_internal/+/master/go/src/infra_internal/puppet/) for Labs/Golo (assuming manual bootstrap of trust by sysadmins, like we do currently with puppet).

For this we'll need:

1. Be able to run go binaries ('cipd', 'puppet_ssl_helper', 'luci_machine_tokend') on the ChromeOS machine. We link them statically for armv6l, so maybe they work out of the box already. Can be confirmed by trying to bootstrap CIPD client via https://chromium.googlesource.com/chromium/tools/depot_tools/+/master/cipd

2. Some sort of cron mechanism, so we can run 'luci_machine_tokend' periodically. It takes static private key and produces short lived authentication token used with Swarming.

Separately, we'll also need 

1. A mechanism to auto-start Swarming bot after reboots.
2. Allow the bot user to reboot the machine.

---

We'll lack some goodies we have from universally using puppet (primarily machine-level monitoring and gnubby-based SSH), but it is probably ok.

Generally, I think treating ChromeOS devices as standalone entities managed externally from Linux (like we do with Android) might be more manageable long-term solution :( Who knows what they change in later versions of ChromeOS. The OS seems pretty tight and we are intruding with our stuff.

Sorry, you're right, and I was too harsh. I agree that if we can get swarming running on-device and don't need the split host/device setup there are a lot of advantages.

What I meant more was that we probably want things either to be baked into the image (which is regularly uprevved by the CrOS team) or versioned and pushed alongside the test (i.e., fetched in an isolate, etc.).

It's possible that we want a Puppet-like solution to manage the tokens we need, since I don't think we want those to be in the base image, and trying to fetch them via the swarming bot may have bootstrapping issues. Puppet's a pretty heavyweight option for that; but I admit that I don't see another great option other than rolling something custom. Maybe we can package the tokens into a separate CIPD package that is properly access-controlled somehow?

Thanks for the pointers Vadim, I'll see how far I can get with each of those requirements. (I certainly don't fully grok the puppet agent cert bootstrapping flow, so I'll ping you if I have questions.)

As for keeping the devices as close to the base image as possible, that's a good point. But I've seen our official "flash server" (build1-a1) we have for these devices. It's simply a series of hand-rolled ssh commands that target each individual chromebook, which is itself running on a nightly cron. (That I've since disabled.) If I can reduce the necessary device bootstrapping into no more than a couple sane ssh commands, tagging them to the end of the flash_all.sh we have running seems perfectly reasonable to me.

And let me again emphasize that this is *experimental*. I'm just testing the waters out here. There's a good chance I'll fallback to running the bot on a separate host if things continue to be as difficult as they are.

Well I got puppet running and auth'ed on the chromebook. It started downloading all the agent libraries from the master, but didn't make it very far before it quit with errors about being an unsupported flavor of unix (ie: not darwin or debian). I was hoping that it would *just work*, but alas :(

Given that adding cros exceptions to all our classes/facters would be a pain/tech-debt nightmare (and any other type of solution would probably run into the same problems when running on bizarro-unix), I think a separate controller is probably the way to go.

So to do that: We've got three r230s sitting idle nearby. If each swarming bot controlled a single device, that'd be 10 bots per server. Container isolations seems like overkill since it's not like a bot would start trying to talk to a device it doesn't own. But since docker appears to be the running-multiple-swarming-bots-on-same-machine solution that we're converging on, we can hijack the usb-docker setup for android. Instead of being assigned a usb port at bot-startup, it'd be assigned an ip address. From there, the swarming bot would need to poke and prod its device over ssh, either quarantining if it's not responding or updating bot state to advertise the dimensions of the device.

All of that's very doable, and shouldn't be terribly difficult given the infrastructure already in place. Will starting working on that next week.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/9c1b632fdbe6fa545422e7322dfdf23e470dbf2e

commit 9c1b632fdbe6fa545422e7322dfdf23e470dbf2e
Author: Elliott Friedman <friedman@google.com>
Date: Tue Jan 30 20:14:17 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/6c9d3cebb115c21349b742f98794ff4374954863

commit 6c9d3cebb115c21349b742f98794ff4374954863
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Tue Jan 30 22:35:53 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/77eb6a489689882a23350fa83c7d750950bb9d07

commit 77eb6a489689882a23350fa83c7d750950bb9d07
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Thu Feb 01 01:16:02 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/5df250555209946d97064526dd5a5ca5c9cd1758

commit 5df250555209946d97064526dd5a5ca5c9cd1758
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Thu Feb 01 03:38:48 2018

Generalize android_docker image building for CrOS as well.

Now running build.sh will create both android_docker and cros_docker
images. Both are similar but have small tweaks to some local files
copied into the image.

The same Dockerfile is used for both, but the context dir changes
for each device type.

Bug:  801235 
Change-Id: I68155eb156d5d920525cbd462c21752da49212b1
Reviewed-on: https://chromium-review.googlesource.com/894741
Commit-Queue: Benjamin Pastene <bpastene@chromium.org>
Reviewed-by: Elliott Friedman <friedman@chromium.org>

[rename] https://crrev.com/5df250555209946d97064526dd5a5ca5c9cd1758/docker/docker_devices/README.md
[rename] https://crrev.com/5df250555209946d97064526dd5a5ca5c9cd1758/docker/docker_devices/Dockerfile
[add] https://crrev.com/5df250555209946d97064526dd5a5ca5c9cd1758/docker/docker_devices/build.sh
[rename] https://crrev.com/5df250555209946d97064526dd5a5ca5c9cd1758/docker/docker_devices/cros/shutdown.sh
[rename] https://crrev.com/5df250555209946d97064526dd5a5ca5c9cd1758/docker/docker_devices/android/start_swarm_bot.sh
[add] https://crrev.com/5df250555209946d97064526dd5a5ca5c9cd1758/docker/docker_devices/cros/start_swarm_bot.sh
[copy] https://crrev.com/5df250555209946d97064526dd5a5ca5c9cd1758/docker/docker_devices/android/shutdown.sh
[delete] https://crrev.com/ed978bda586620576fa27c754e9574b9139bfe70/docker/android_devices/build.sh

I've got a working set of cros container bots on staging:
https://chromium-swarm-dev.appspot.com/botlist?c=id&c=os&c=task&c=status&f=os%3AChromeOS&l=100&q=os%3A&s=id%3Aasc

A host (build33-a1) has been assigned 10 cros devices (build{22.31}-a1). For each device, a container is running on the host. Each container has a swarming bot running that's responsible for its device. A few notes so far:

- Bot names are ugly. I'm working on fixing those up.

- Outside the containers, the linux host gets its list of devices to manage via a json file on the host, the contents of which are deployed via puppet: (thanks to friedman for coming with a pretty clean way of managing that)
https://chrome-internal.googlesource.com/infra/puppet/+/95b55a4fdd1f7b33457b98f1defa4a45ed3a0404/puppetm/etc/puppet/hieradata/default.yaml#519

- Inside the container, the swarming bot gets the hostname of the bot it manages via env var. (I think I called it SWARMING_CROS_HOSTNAME or somesuch.)

- All device poking/interaction (getting device state/dimensions) is done via ssh.

- On container startup, we gen some ssh keys. On bot startup, we register the pub key on the device. That, plus some specific options in ~/.ssh/config, enables a simple format for any future ssh commands to the device (ie: `ssh root@$SWARMING_CROS_HOSTNAME uname`). Consequently, actual tasks don't have to worry about ssh passwords/auth.

- Quarantining on unavailable devices even works! https://chromium-swarm-dev.appspot.com/bot?id=build33-a1--build27-a1

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/1ebe658ce72fd2dbb7619d004fdd9c6b78230ea3

commit 1ebe658ce72fd2dbb7619d004fdd9c6b78230ea3
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Fri Feb 02 01:30:14 2018

Gen some ssh keys on cros container startup.

R=friedman@chromium.org

Bug:  801235 
Change-Id: If1a2c9ec21452d0cf4fee72d9ecb378bf516e1bf
Reviewed-on: https://chromium-review.googlesource.com/898402
Reviewed-by: Elliott Friedman <friedman@chromium.org>
Commit-Queue: Benjamin Pastene <bpastene@chromium.org>

[modify] https://crrev.com/1ebe658ce72fd2dbb7619d004fdd9c6b78230ea3/docker/docker_devices/cros/start_swarm_bot.sh

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/6522309da3a98aebddf3d0fd8fb1ab82bfba0121

commit 6522309da3a98aebddf3d0fd8fb1ab82bfba0121
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Fri Feb 16 00:42:07 2018

Add a service for running CrOS swarming bots inside docker containers.

Similar to android_docker, cros_docker will wrap a swarming bot running
on a linux host that pairs itself to a device.

Bug:  801235 
Change-Id: If46208767a399b95be4fc1bdc5ca22d033f0fa4f
Reviewed-on: https://chromium-review.googlesource.com/896612
Commit-Queue: Benjamin Pastene <bpastene@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>

[add] https://crrev.com/6522309da3a98aebddf3d0fd8fb1ab82bfba0121/infra/services/cros_docker/host.py
[modify] https://crrev.com/6522309da3a98aebddf3d0fd8fb1ab82bfba0121/infra/services/swarm_docker/test/containers_test.py
[add] https://crrev.com/6522309da3a98aebddf3d0fd8fb1ab82bfba0121/infra/services/cros_docker/test/host_test.py
[add] https://crrev.com/6522309da3a98aebddf3d0fd8fb1ab82bfba0121/infra/services/cros_docker/containers.py
[modify] https://crrev.com/6522309da3a98aebddf3d0fd8fb1ab82bfba0121/infra/services/android_docker/containers.py
[add] https://crrev.com/6522309da3a98aebddf3d0fd8fb1ab82bfba0121/infra/services/cros_docker/__init__.py
[add] https://crrev.com/6522309da3a98aebddf3d0fd8fb1ab82bfba0121/infra/services/cros_docker/test/__init__.py
[add] https://crrev.com/6522309da3a98aebddf3d0fd8fb1ab82bfba0121/infra/services/cros_docker/.coveragerc
[add] https://crrev.com/6522309da3a98aebddf3d0fd8fb1ab82bfba0121/infra/services/cros_docker/test/containers_test.py
[add] https://crrev.com/6522309da3a98aebddf3d0fd8fb1ab82bfba0121/infra/services/cros_docker/__main__.py
[modify] https://crrev.com/6522309da3a98aebddf3d0fd8fb1ab82bfba0121/infra/services/swarm_docker/containers.py

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chrome/tools/build/+/28feae53e6a93ca1ef6c42cf1c51971ba025763e

commit 28feae53e6a93ca1ef6c42cf1c51971ba025763e
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Fri Feb 16 03:11:53 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/master-manager/+/8c031b19a9216d32673694107553e74df501de75

commit 8c031b19a9216d32673694107553e74df501de75
Author: Benjamin Pastene <bpastene@google.com>
Date: Fri Feb 16 03:35:35 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/master-manager/+/8c031b19a9216d32673694107553e74df501de75

commit 8c031b19a9216d32673694107553e74df501de75
Author: Benjamin Pastene <bpastene@google.com>
Date: Fri Feb 16 03:35:35 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/c9f2ccc01e79b6ac3c9d391afd02f9ddefddb364

commit c9f2ccc01e79b6ac3c9d391afd02f9ddefddb364
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Fri Feb 16 18:51:18 2018

android_docker: Fix path to Dockerfile in build script.

TBR=friedman@chromium.org
Bug:  801235 
Change-Id: I1da7a43bf461a5adc58e2b02f4028998515b1659
Reviewed-on: https://chromium-review.googlesource.com/924278
Reviewed-by: Benjamin Pastene <bpastene@chromium.org>
Commit-Queue: Benjamin Pastene <bpastene@chromium.org>

[modify] https://crrev.com/c9f2ccc01e79b6ac3c9d391afd02f9ddefddb364/docker/docker_devices/build.sh

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/dfbe8ca4ddfd59befc71bf82c90b4f46b63c54c8

commit dfbe8ca4ddfd59befc71bf82c90b4f46b63c54c8
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Fri Feb 16 21:23:46 2018

docker devices: Duplicate Dockerfiles as a workaround bug in docker.

Revert this once crbug.com/813179 is done.

TBR=friedman@chromium.org
Bug:  801235 
Change-Id: Iff601133a554e91a0cb6b5b229627fe8f8218996
Reviewed-on: https://chromium-review.googlesource.com/923434
Reviewed-by: Benjamin Pastene <bpastene@chromium.org>
Commit-Queue: Benjamin Pastene <bpastene@chromium.org>

[add] https://crrev.com/dfbe8ca4ddfd59befc71bf82c90b4f46b63c54c8/docker/docker_devices/android/Dockerfile
[add] https://crrev.com/dfbe8ca4ddfd59befc71bf82c90b4f46b63c54c8/docker/docker_devices/cros/Dockerfile
[modify] https://crrev.com/dfbe8ca4ddfd59befc71bf82c90b4f46b63c54c8/docker/docker_devices/build.sh

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/489dd0e2506652237e4b857e11acd6d560965ed2

commit 489dd0e2506652237e4b857e11acd6d560965ed2
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Fri Feb 16 22:50:16 2018

Rename infra/android_docker CIPD package to infra/docker_devices.

So that it can be used for android and cros devices.

Bug:  801235 
Change-Id: I881e05b182cbeaeadcbd41aaf350c784be63aef3
Reviewed-on: https://chromium-review.googlesource.com/923380
Reviewed-by: Vadim Shtayura <vadimsh@chromium.org>
Commit-Queue: Benjamin Pastene <bpastene@chromium.org>

[rename] https://crrev.com/489dd0e2506652237e4b857e11acd6d560965ed2/build/packages/docker_devices.yaml

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/d3280bc6ac47162a29e4a89fcb44af2752bb8d8b

commit d3280bc6ac47162a29e4a89fcb44af2752bb8d8b
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Sat Feb 17 00:20:24 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/ce8bb91e53afe4250bec370ed786d72c1918a881

commit ce8bb91e53afe4250bec370ed786d72c1918a881
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Sat Feb 17 00:41:14 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/a54734bab407e3877d744d62c2c4f3100397abd5

commit a54734bab407e3877d744d62c2c4f3100397abd5
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Sat Feb 17 00:56:55 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/499a71e62d6ecffced0ed56b8b77af4d7d74f9c0

commit 499a71e62d6ecffced0ed56b8b77af4d7d74f9c0
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Sat Feb 17 01:26:25 2018

Got the docker deployment finished. All 30 cros devices now show up in swarming (minus a few that are dead and prob need healing):
https://chromium-swarm-dev.appspot.com/botlist?c=id&c=os&c=task&c=status&f=os%3AChromeOS&l=100&q=os%3AC&s=id%3Aasc

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/3e876102485547df373df080915caa500c2ec981

commit 3e876102485547df373df080915caa500c2ec981
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Wed Feb 21 22:32:22 2018

Devices seem pretty stable. I'll work on spinning up a bot that launches continuous telemetry unit tests on them. Filed bug 814935 for that

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/a2eff431c9a17b7efe24018cfb54d0f4b1043aae

commit a2eff431c9a17b7efe24018cfb54d0f4b1043aae
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Tue Feb 27 21:59:30 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/3d68dab285bf2015c4d9df7d7fc42df48cd40634

commit 3d68dab285bf2015c4d9df7d7fc42df48cd40634
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Wed Mar 07 23:04:36 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/980d90509bc14471673b9e9a59a61e0e7234609d

commit 980d90509bc14471673b9e9a59a61e0e7234609d
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Mon Mar 12 23:31:02 2018

cros_docker: Point a universal hostname to each container's device.

This will let all tests (and even the swarming bot itself) refer to the
cros_device as "variable_chromeos_device_hostname", which resolves to
the container's device.

Bug:  801235 
Change-Id: I4810767f656b3bdb3cae4b0d3b8d403967001519
Reviewed-on: https://chromium-review.googlesource.com/954054
Commit-Queue: Benjamin Pastene <bpastene@chromium.org>
Reviewed-by: John Budorick <jbudorick@chromium.org>

[modify] https://crrev.com/980d90509bc14471673b9e9a59a61e0e7234609d/infra/services/cros_docker/test/containers_test.py
[modify] https://crrev.com/980d90509bc14471673b9e9a59a61e0e7234609d/infra/services/cros_docker/containers.py

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/c4a89743eea3ff400a802e50af9730745edd1cb7

commit c4a89743eea3ff400a802e50af9730745edd1cb7
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Mon Mar 12 23:42:45 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra/+/e7ead79e1343f7d9a0dbb78ccc087d1af8e9d9fa

commit e7ead79e1343f7d9a0dbb78ccc087d1af8e9d9fa
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Tue Mar 13 18:48:03 2018

cros_docker: Make ssh's UserKnownHostsFile unique per container.

TBR=jbudorick@chromium.org

Bug:  801235 
Change-Id: I20d3900dcef211d6b0b939289ca201b03238d06e
Reviewed-on: https://chromium-review.googlesource.com/961127
Reviewed-by: Benjamin Pastene <bpastene@chromium.org>
Commit-Queue: Benjamin Pastene <bpastene@chromium.org>

[modify] https://crrev.com/e7ead79e1343f7d9a0dbb78ccc087d1af8e9d9fa/infra/services/cros_docker/host.py

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/7c6742e228776089b8af08f4b1d08dc7865ae6da

commit 7c6742e228776089b8af08f4b1d08dc7865ae6da
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Fri Mar 16 19:11:20 2018

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infradata/config/+/e354fcb63f0f85ba5204d5a325c312a01e6585e3

commit e354fcb63f0f85ba5204d5a325c312a01e6585e3
Author: Benjamin Pastene <bpastene@chromium.org>
Date: Fri Mar 23 17:01:11 2018

This is for the most part done. We've got the 30 kevins in swarming taking tasks:
https://chromium-swarm-dev.appspot.com/botlist?c=id&c=os&c=task&c=status&f=os%3AChromeOS&l=100&s=id%3Aasc

Though the tests themselves need some work (tracking in bug 814935)

Since we can theoretically start arbitrarily throwing more cros hardware into swarming, I'm closing this out.