only two moblabs had the 'os:cros' label (chromeos2-row2-rack8-host1,chromeos2-row2-rack8-host3). I removed it from both and we should not see the issue anymore. 

Next steps is to update the factory to create a moblab host using the os labels.

Just for posterity: 

The change to create hosts based off of os labels went in weeks ago but we hit this problem today because the 2 moblab duts had 'os:cros' labels created on them (most likely because they were recently added and as part of host creation got the os labels added to them).  Looking at the dut history, I see that the first job they ever ran was on 8/1 23:59 which seems about right since the paladin started failing with the error starting with this build:

https://uberchromegw.corp.google.com/i/chromeos/builders/guado_moblab-paladin/builds/3360

The previous failures were due to autotest not responding back in time with the run_suite request.

How do we plan to prevent this in the future?

with https://chromium-review.googlesource.com/#/c/365236/

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/410b3cfe62414d0f21fe0a2140a52b2106673dec

commit 410b3cfe62414d0f21fe0a2140a52b2106673dec
Author: Kevin Cheng <kevcheng@chromium.org>
Date: Tue Aug 02 17:21:52 2016

[autotest] Update factory to create moblab host based on os:moblab.

BUG= chromium:633616 
TEST=Locally verified moblab host gets created with os:moblab label.

Change-Id: Id7f14ff8a6800da5121ce344590f77df2abe87c2
Reviewed-on: https://chromium-review.googlesource.com/365236
Commit-Ready: Kevin Cheng <kevcheng@chromium.org>
Tested-by: Kevin Cheng <kevcheng@chromium.org>
Reviewed-by: Dan Shi <dshi@google.com>

[modify] https://crrev.com/410b3cfe62414d0f21fe0a2140a52b2106673dec/server/hosts/moblab_host.py
[modify] https://crrev.com/410b3cfe62414d0f21fe0a2140a52b2106673dec/server/hosts/factory.py