First step in the cleanup is to add a hack to Autotest
code to touch the OOBE file prior to requesting update
on a beaglebone.

Looking through logs, I think I know why beaglebones fail to
update, but the regular DUTs get along just fine.

For regular DUT updates, we use this command line:
    update_engine_client --update --omaha_url=...

For beaglebone, we use this command line instead:
    update_engine_client --check_for_update --omaha_url=

So, the problem is that --check_for_update is broken.

The fix is to drop an is_oobe_enabled=false in the /etc/update_manager.conf for beaglebone. See JS overlay for an example. If you don't have an OOBE flow, then you likely want that.

Re #3:
"--update" means that we are provisioning the device.
"--check_for_update" means that you want to simulate an update check, either interactive or not, and basically triggers the code under test.

If your goal is to test whether "updates work" in beaglebone (probably not the case) then use --check_for_update. If you are provisioning then use --update.

Also, you can recover all existing devices by adding deadline="now" to the omaha response in the devserver or omaha config (I don't know where are you serving these updates from)

Regarding the difference between --update and --check_for_update:
The behavior is inexplicable.  Reading the source, it seems that
"--update" is equivalent to "--check_for_update --follow".  So,
I can't explain why they should behave differently.  That said, I
have anecdotal evidence that it's so:  I've successfully updated
a beaglebone from 8281 to 8316 at my desktop, without difficulty
using --update.

More experiments to come.

As I said, --update means provisioning and therefore it should skip this OOBE checking because you are not trying to test it. --check_for_update will instead not update because the update is not critical (no deadline passed) and OOBE was not completed.

--update is not equivalent to --check_for_update --folow; it also sets the app_version to "ForcedUpdate" with flags the daemon that it should get an update anyway.

See: 
https://android.googlesource.com/platform/system/update_engine/+/master/update_engine_client.cc#445

I've confirmed the behavior as described:  I ran update_engine_client
with --check_for_update, and the status went immediately back to IDLE.
Then I retried with --update instead on the same beaglebone, and now
it the status is DOWNLOADING.

> --update is not equivalent to --check_for_update --folow;
> it also sets the app_version to "ForcedUpdate" with flags
> the daemon that it should get an update anyway.

Aha!  I missed that trick the first go around:
  if (do_update_request) {
    LOG_IF(WARNING, FLAGS_reboot) << "-reboot flag ignored.";
    string app_version = FLAGS_app_version;
    if (FLAGS_update && app_version.empty()) {
      app_version = "ForcedUpdate";
      LOG(INFO) << "Forcing an update by setting app_version to ForcedUpdate.";
    }
    LOG(INFO) << "Initiating update check and install.";
    if (!client_->AttemptUpdate(app_version, FLAGS_omaha_url,
                                FLAGS_interactive)) {
      LOG(ERROR) << "Error checking for update.";
      return 1;
    }
  }

"do_update_request" includes both --update and --check_for_update,
but the selection of ForcedUpdate only applies to --update.

OK.  Mostly, I understand the problem.  I'll let the correct fix
percolate.

There was a brief time where the --update would also hit the kNonCriticalUpdateInOOBE error, I don't remember if that hit canary or your build. cernekee@ fixed it. Given that you tested it on your build you probably don't have that problem.

Adding deadline=now to the response for beaglebone makes the update apply in all cases; touching ~/.oobe_complete file as suggested in #2 will also work in all cases.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/7e53aa05c0c24dac04760d3cb5a68a8e5c708d6a

commit 7e53aa05c0c24dac04760d3cb5a68a8e5c708d6a
Author: Richard Barnette <jrbarnette@chromium.org>
Date: Fri May 20 17:49:40 2016

[autotest] Hack to allow servos in the lab to update.

A recent update_engine change means that beaglebones in the lab
are now refusing to update unless they've gone through OOBE.
So, touch /home/chronos/.oobe_completed, so we can get them
updated.

BUG= chromium:613603 
TEST=None

Change-Id: I7c805fe33902bf903bfc571f5bdd3b9e5dcdd245
Reviewed-on: https://chromium-review.googlesource.com/346225
Commit-Ready: Richard Barnette <jrbarnette@chromium.org>
Tested-by: Richard Barnette <jrbarnette@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>

[modify] https://crrev.com/7e53aa05c0c24dac04760d3cb5a68a8e5c708d6a/server/hosts/servo_host.py

This problem is _so_ fixed, I can't even begin.

Post-mortem is here:
    https://docs.google.com/document/d/1jkmIvX4xZfZyA4DyHWun-u7Ua1PY4OvV_gCLK1dzkqg/edit#