Looking at scheduler logs and looking at what we have on monarch, I think this is actually higher priority than I thought.

06/27 18:22:11.660 ERROR|         interface:0143| Automatic monitoring flush failed.
Traceback (most recent call last):
  File "/usr/local/autotest/site-packages/chromite/third_party/infra_libs/ts_mon/common/interface.py", line 141, in _flush_and_log_exceptions
    flush()
  File "/usr/local/autotest/site-packages/chromite/third_party/infra_libs/ts_mon/common/interface.py", line 89, in flush
    metric.serialize_to(proto, start_time, fields, value, target)
  File "/usr/local/autotest/site-packages/chromite/third_party/infra_libs/ts_mon/common/metrics.py", line 101, in serialize_to
    self._populate_fields(metric_pb, fields)
  File "/usr/local/autotest/site-packages/chromite/third_party/infra_libs/ts_mon/common/metrics.py", line 128, in _populate_fields
    raise errors.MonitoringInvalidFieldTypeError(self._name, key, value)
MonitoringInvalidFieldTypeError: Metric "chromeos/autotest/dut_working" was given field "board" with invalid value "None" (<type 'NoneType'>).


What this suggests to me is that this is failing during the flush, and that once it fails we are hosed as far as ts_mon on that process. That explains why we are still seeing so little data in monarch.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/8aa71653676d6ad54620b6ef8fa10b8785b29454

commit 8aa71653676d6ad54620b6ef8fa10b8785b29454
Author: Paul Hobbs <phobbs@google.com>
Date: Wed Jun 29 18:59:01 2016

[autotest] Default board='' for dut_working metric

If board is None, it causes ts_mon to throw an exception and fail to
send stats until the scheduler process is restarted. This change sets
board to be the empty string when missing.

BUG=chromium:624051,  chromium:623299 
TEST=None

Change-Id: If908dc9ebe065c272ecf04fe71a66c961d859520
Reviewed-on: https://chromium-review.googlesource.com/357180
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Tested-by: Paul Hobbs <phobbs@google.com>

[modify] https://crrev.com/8aa71653676d6ad54620b6ef8fa10b8785b29454/scheduler/rdb_hosts.py

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/8aa71653676d6ad54620b6ef8fa10b8785b29454

commit 8aa71653676d6ad54620b6ef8fa10b8785b29454
Author: Paul Hobbs <phobbs@google.com>
Date: Wed Jun 29 18:59:01 2016

[autotest] Default board='' for dut_working metric

If board is None, it causes ts_mon to throw an exception and fail to
send stats until the scheduler process is restarted. This change sets
board to be the empty string when missing.

BUG=chromium:624051,  chromium:623299 
TEST=None

Change-Id: If908dc9ebe065c272ecf04fe71a66c961d859520
Reviewed-on: https://chromium-review.googlesource.com/357180
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Tested-by: Paul Hobbs <phobbs@google.com>

[modify] https://crrev.com/8aa71653676d6ad54620b6ef8fa10b8785b29454/scheduler/rdb_hosts.py

Bulk verified