I'd prefer a centralized solution instead of writing local device files.

Can you elaborate more on your monarch idea? I guess one solution would be to just have a 'check-in' ping whenever we see the device, and do a monarch query for devices that haven't checked in for a while. We would have to report the host the device is on as well. This has the disadvantage that decommissioned devices would keep showing up unless we had some way to remove them from the data.

Ultimately it would be awesome to have a machine database to handle this kind of stuff, but that seems a bit far off at the moment.

Regarding the monarch solution: that's correct. Any device that's stopped reporting its status would be assumed offline.

I agree a centralized database of this information would be best, but that's still a ways off from being usable and we're running into issues with missing devices today. Ditto with the ts_mon stuff.

I vote for adding a local file to the host. It'll be a quick and small hack that can easily be undone when one of these other solutions becomes available. 

It's important to discern between devices genuinely disconnected and dead devices (e.g. powered off).

What I'd like to know is what lsusb reports on unpowered / hung devices, so that ghost devices (e.g. down but still connected via usb) could still be reported appropriately.

Then the problem is trivial to fix as the swarming bot can report itself as Android properly.

IIRC lsusb reports nothing for unpowered / hung devices. (We see a separate class of issues in which lsusb reports the device but adb doesn't.)

Oh, that's sad to hear. :(
I was specifically thinking about "lsusb reports the device but adb doesn't" which could be raised up in swarming bot code but thining about it more, python-adb probably uses the same basis as lsusb, so it (probably) already does the same output for unavailable devices.

So I now realize that in comment 1 bpastene@ was referring to the existing monitoring / ticket filing pipeline built in https://bugs.chromium.org/p/chromium/issues/detail?id=519884. The advantage there is that the query-monarch-and-file-a-ticket logic is already implemented in https://cs.corp.google.com/piper///depot/google3/googleclient/chrome/infra/device_ticket_filer/device_ticket_filer.py.

I'll take ownership of this bug for now.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/infra_internal.git/+/9b3180106d2a000dc2bdb1da173de965ea6437fb

commit 9b3180106d2a000dc2bdb1da173de965ea6437fb
Author: stip <stip@google.com>
Date: Thu May 12 21:23:54 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/2a563c158ad5cb890b6d4ea008b7c24ce67d2db0

commit 2a563c158ad5cb890b6d4ea008b7c24ce67d2db0
Author: bpastene <bpastene@chromium.org>
Date: Tue May 17 22:22:45 2016

Delete devicemon; moving to sysmon.

BUG= 608427 
TBR=stip@chromium.org

Review-Url: https://codereview.chromium.org/1993443002

[delete] https://crrev.com/60bad66d69233bed21c884641796c17f3a72f840/infra/services/devicemon/__init__.py
[delete] https://crrev.com/60bad66d69233bed21c884641796c17f3a72f840/infra/services/devicemon/__main__.py
[delete] https://crrev.com/60bad66d69233bed21c884641796c17f3a72f840/infra/services/devicemon/device_metrics.py
[delete] https://crrev.com/60bad66d69233bed21c884641796c17f3a72f840/infra/services/devicemon/test/__init__.py
[delete] https://crrev.com/60bad66d69233bed21c884641796c17f3a72f840/infra/services/devicemon/test/device_metrics_test.py

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/a6f6037e0f6bc697b331bf1c0b54ec77ed634399

commit a6f6037e0f6bc697b331bf1c0b54ec77ed634399
Author: stip <stip@chromium.org>
Date: Fri May 20 22:48:41 2016

Send ts_mon metrics from android device data.

BUG= 608427 

Review-Url: https://codereview.chromium.org/1981853002

[modify] https://crrev.com/a6f6037e0f6bc697b331bf1c0b54ec77ed634399/infra/services/sysmon/__main__.py
[add] https://crrev.com/a6f6037e0f6bc697b331bf1c0b54ec77ed634399/infra/services/sysmon/android_device_metrics.py
[add] https://crrev.com/a6f6037e0f6bc697b331bf1c0b54ec77ed634399/infra/services/sysmon/test/android_device_metrics_test.py
[add] https://crrev.com/a6f6037e0f6bc697b331bf1c0b54ec77ed634399/infra/services/sysmon/test/data/invalid_files/invalid_json_android_device_status.json
[add] https://crrev.com/a6f6037e0f6bc697b331bf1c0b54ec77ed634399/infra/services/sysmon/test/data/invalid_files/invalid_version_android_device_status.json
[add] https://crrev.com/a6f6037e0f6bc697b331bf1c0b54ec77ed634399/infra/services/sysmon/test/data/invalid_files/not_dict_android_device_status.json
[add] https://crrev.com/a6f6037e0f6bc697b331bf1c0b54ec77ed634399/infra/services/sysmon/test/data/valid_files/android_device_status.json
[add] https://crrev.com/a6f6037e0f6bc697b331bf1c0b54ec77ed634399/infra/services/sysmon/test/data/valid_files/no_temp_android_device_status.json

We're getting data now from the swarming devices. Only thing left to do is get the queries from monarch to locate devices that have dropped off.

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/07a8e221f3b07bbbd93c97952ea3e06ce21e0dd4

commit 07a8e221f3b07bbbd93c97952ea3e06ce21e0dd4
Author: Mike Stipicevic <stip@chromium.org>
Date: Fri May 20 23:39:25 2016

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/infra/puppet/+/e53317158e1a8c50ddedee33f2d2025544f26794

commit e53317158e1a8c50ddedee33f2d2025544f26794
Author: Mike Stipicevic <stip@chromium.org>
Date: Sat May 21 00:06:57 2016

The following revision refers to this bug:
  https://chromium.googlesource.com/infra/infra.git/+/316fd863534efaa772b0d333cd7e57dea181506a

commit 316fd863534efaa772b0d333cd7e57dea181506a
Author: bpastene <bpastene@chromium.org>
Date: Thu Jun 02 00:42:04 2016

Don't report android device port paths.

BUG= 608427 

Review-Url: https://codereview.chromium.org/2023003004

[modify] https://crrev.com/316fd863534efaa772b0d333cd7e57dea181506a/infra/services/sysmon/android_device_metrics.py
[modify] https://crrev.com/316fd863534efaa772b0d333cd7e57dea181506a/infra/services/sysmon/test/android_device_metrics_test.py
[add] https://crrev.com/316fd863534efaa772b0d333cd7e57dea181506a/infra/services/sysmon/test/data/valid_files/some_port_paths.json