Thanks for the analysis - looking into it now.

From looking at var/log/messages, it looks like arc-setup has crashed:
https://pantheon.corp.google.com/storage/browser/chromeos-autotest-results/168579152-chromeos-test/chromeos2-row4-rack4-host3/cheets_StartAndroid.stress/sysinfo/var/log

2018-01-08T17:48:00.013577+00:00 WARNING crash_reporter[1509]: Could not load the device policy file.
2018-01-08T17:48:00.013748+00:00 WARNING crash_reporter[1509]: [user] Received crash notification for arc-setup[1494] sig 6, user 0 (developer build - not testing - always dumping)

And looking at /var/log/arc-setup.log:
[0108/095118:INFO:arc_setup.cc(531)] Running !LaunchAndWait({generate_camera_profile.value()})...

Looks like something is wrong with the camera profile for this particular board?

Sorry, here's a better look at arc-setup part of the log:

[0108/095118:INFO:arc_setup.cc(531)] Running !LaunchAndWait({generate_camera_profile.value()})...
[FATAL:at_exit.cc(54)] Check failed: false. Tried to RegisterCallback without an AtExitManager
/usr/lib64/libbase-core-395517.so(base::debug::StackTrace::StackTrace()+0x13) [0x7b2a6bc52ef3]

[0108/095118:FATAL:arc_setup.cc(531)] Check failed: !(!LaunchAndWait({generate_camera_profile.value()})). 
/usr/lib64/libbase-core-395517.so(base::debug::StackTrace::StackTrace()+0x13) [0x7fda5997eef3]

Henry I see that you added the check for camera profile. Any ideas?

This is bizarre. I just checked both coral-paladin and coral-release, which both were R65-10291.0.0. But, coral-paladin is failing, while coral-release is passing for this particular test.

paladin: http://cautotest-prod.corp.google.com/tko/compose_query.cgi?rows=hostname&condition=tag~%27168640336-%25%27%26test%3D%27cheets_StartAndroid.stress%27&columns=label&title=Report
release: http://cautotest-prod.corp.google.com/tko/compose_query.cgi?rows=hostname&condition=tag~%27168605498-%25%27%26test%3D%27cheets_StartAndroid.stress%27&columns=label&title=Report

According to bhthompson@ it should not be the case that these two test cases return different results, especially when I am checking the same test suite (bvt-inline).

For camera profile, it depends on arc-camera-profile and the ebuild is merged.
generate tool:
https://chromium-review.googlesource.com/c/chromiumos/platform/arc-camera/+/836648

generate tool ebuild:
https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/838861

dynamic ebuild depends on arc-camera-profile
https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/841905

coral uses arc-media-profiles-dynamic
https://chrome-internal-review.googlesource.com/#/c/chromeos/overlays/overlay-coral-private/+/533079/

Not sure why paladin build doesn't install it.

https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/coral-paladin/R65-10290.0.0-rc3?pli=1
The image has /usr/bin/generate_camera_profile binary.

But when I executed the binary, there is an error:
localhost lib64 # /usr/bin/generate_camera_profile 
[FATAL:at_exit.cc(54)] Check failed: false. Tried to RegisterCallback without an AtExitManager
/usr/lib64/libbase-core-395517.so(base::debug::StackTrace::StackTrace()+0x13) [0x7e8b0adb0ef3]

Aborted (core dumped)

https://chromium-review.googlesource.com/c/chromiumos/platform/arc-camera/+/856382

I upload a CL to fix it.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/46c6e190a528e52cfae9ecad3fc6a9e698575d19

commit 46c6e190a528e52cfae9ecad3fc6a9e698575d19
Author: Heng-Ruey Hsu <henryhsu@chromium.org>
Date: Tue Jan 09 13:49:38 2018

tool: Add AtExitManager

We need AtExitManager. Otherwise it failed on AtExitManager::RegisterTask()

BUG= chromium:799997 
TEST=cros deploy arc-camera-profile and run the binary without error

Change-Id: I56dfa3a7f1d6e2ac98e16c24913b834d3e50db90
Reviewed-on: https://chromium-review.googlesource.com/856382
Commit-Ready: Heng-ruey Hsu <henryhsu@chromium.org>
Tested-by: Heng-ruey Hsu <henryhsu@chromium.org>
Reviewed-by: Heng-ruey Hsu <henryhsu@chromium.org>

[modify] https://crrev.com/46c6e190a528e52cfae9ecad3fc6a9e698575d19/tools/generate_camera_profile.cc

Thanks Henry! Looks like coral-paladin is green now:
https://uberchromegw.corp.google.com/i/chromeos/builders/coral-paladin/builds/2247

Anyone know if this same root cause was responsible for  bug #799948  and  bug #799958  ?

I took a look at those 2 bugs and they don't have the same error stack, so I don't believe so. I've updated the two bugs.

@13: I don't know the general policy, but in general I think we actually care about investigating flake bugs.  We have so many builders that if we let flake bugs go unfixed then the build is always failing for one flake or another...

For now, I'm re-opening the bugs but someone can correct me if I'm wrong.

Actually re-opening, per c#14, and also p er  crbug.com/799958#c3 

Right.  _This_ one is fixed.  The other two need to be understood or sit there for quite a while without reproducing before being closed.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/f8463eb078ca35670cd9ebefcfff8c3b6b4702a3

commit f8463eb078ca35670cd9ebefcfff8c3b6b4702a3
Author: Heng-Ruey Hsu <henryhsu@chromium.org>
Date: Fri Jan 19 14:04:50 2018

tool: Add AtExitManager

We need AtExitManager. Otherwise it failed on AtExitManager::RegisterTask()

BUG= chromium:799997 
TEST=cros deploy arc-camera-profile and run the binary without error

Change-Id: Ic95826891008fce5fea5e22fec404ca649f3e521
Reviewed-on: https://chromium-review.googlesource.com/856382
Commit-Ready: Heng-ruey Hsu <henryhsu@chromium.org>
Tested-by: Heng-ruey Hsu <henryhsu@chromium.org>
Reviewed-by: Heng-ruey Hsu <henryhsu@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/875638
Reviewed-by: Ricky Liang <jcliang@chromium.org>
Tested-by: Ricky Liang <jcliang@chromium.org>
Commit-Queue: Ricky Liang <jcliang@chromium.org>

[modify] https://crrev.com/f8463eb078ca35670cd9ebefcfff8c3b6b4702a3/tools/generate_camera_profile.cc