The container was up and running because /var/log/messages has

  2018-05-17T13:07:51.085467+00:00 INFO session_manager[15859]: [INFO:upstart_signal_emitter.cc(35)] Emitting arc-booted Upstart signal

(which means the container reached the boot_progress_enable_screen phase and the mojo connection was working), and android.kmsg, arc-lifetime.log, and /var/run/android/ (run_oci log) don't say container was shut down. So it's safe to say the container was up, but for some reason adb didn't work.

android.kmsg has 

  [  667.862115] <5>init: Starting service 'adbd'...
  <EOF>

so the container at least tried to start it, but that's all I can tell. logcat logs seem missing which is also strange and because of that, it's hard to tell what happened in the container.

Luis, since this is adb, do you have any idea why this happened?

No idea, but maybe the way we are checking some random logcat output for a string is Considered Harmful™. I'll add some more logging to begin with.

As to why logcat is missing, that's because cheets_CleanShutDown is not an arc.ArcTest, so logcat collection is not enabled :(

Sent a few patches to improve this situation, and to have better logs. I guess I'll take the bug.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/3e6a9350390e555c7c63f0a14fe8c18cf05a7a81

commit 3e6a9350390e555c7c63f0a14fe8c18cf05a7a81
Author: Luis Hector Chavez <lhchavez@google.com>
Date: Sat May 19 00:19:53 2018

arc: Make the adb connection a bit more robust

This change:

* Stops needlessly tying some random container state to adb connection
  being ready. Instead, just connect!
* Adds logging if the connection cannot be established on time.

BUG= chromium:844097 
TEST=test_that ${DUT} cheets_ContainerSmokeTest

Change-Id: I8a2ef316d72fad8dac5a89c51861080352b8bedb
Reviewed-on: https://chromium-review.googlesource.com/1064548
Commit-Ready: Luis Hector Chavez <lhchavez@chromium.org>
Tested-by: Luis Hector Chavez <lhchavez@chromium.org>
Reviewed-by: Yusuke Sato <yusukes@chromium.org>
Reviewed-by: Victor Hsieh <victorhsieh@chromium.org>

[modify] https://crrev.com/3e6a9350390e555c7c63f0a14fe8c18cf05a7a81/client/common_lib/cros/arc.py

We have seen exactly zero failures of this kind since the fix was merged: https://stainless.corp.google.com/search?view=matrix&row=model&col=build&first_date=2018-05-19&last_date=2018-05-22&test=%5Echeets%5C_CleanShutDown%24&status=GOOD&status=FAIL&exclude_cts=false&exclude_not_run=true&exclude_non_release=true&exclude_au=true&exclude_acts=true&exclude_retried=true&exclude_non_production=false

declaring victory.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/third_party/autotest/+/b740ab9dbc7ff362970ad8203e5dec1ce702fa5d

commit b740ab9dbc7ff362970ad8203e5dec1ce702fa5d
Author: Luis Hector Chavez <lhchavez@google.com>
Date: Wed May 23 19:50:50 2018

arc: Extract the logcat collection logic

This change adds a Logcat context manager, that can be used in
non-arc.ArcTest tests.

BUG= chromium:844097 
TEST=cheets_ContainerSmokeTest

Change-Id: I52271a84ce69bea867658424173fdf88a966d718
Reviewed-on: https://chromium-review.googlesource.com/1064549
Commit-Ready: Luis Hector Chavez <lhchavez@chromium.org>
Tested-by: Luis Hector Chavez <lhchavez@chromium.org>
Reviewed-by: Luis Hector Chavez <lhchavez@chromium.org>
Reviewed-by: Victor Hsieh <victorhsieh@chromium.org>

[modify] https://crrev.com/b740ab9dbc7ff362970ad8203e5dec1ce702fa5d/client/common_lib/cros/arc.py

The following revision refers to this bug:
  https://chrome-internal.googlesource.com/chromeos/autotest-cheets/+/f9c79b0954a589061f31cec8c0f03d43cfd1e64c

commit f9c79b0954a589061f31cec8c0f03d43cfd1e64c
Author: Luis Hector Chavez <lhchavez@google.com>
Date: Wed May 30 00:15:24 2018

 Issue 843118  has been merged into this issue.