From https://luci-logdog.appspot.com/v/?s=chromeos/bb/chromeos/bob-release/1570/+/recipes/steps/UnitTest/0/stdout

metrics-0.0.1-r2805: >>> Test phase: chromeos-base/metrics-0.0.1-r2805
metrics-0.0.1-r2805: /build/bob/tmp/portage/chromeos-base/metrics-0.0.1-r2805/work/metrics-0.0.1/platform2/common-mk/platform2_test.py --action=pre_test --sysroot=/build/bob -- 
metrics-0.0.1-r2805: ERROR: ld.so: object 'libsandbox.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
metrics-0.0.1-r2805: /build/bob/tmp/portage/chromeos-base/metrics-0.0.1-r2805/work/metrics-0.0.1/platform2/common-mk/platform2_test.py --action=run --sysroot=/build/bob -- /build/bob/var/cache/portage/chromeos-base/metrics/out/Default/cumulative_metrics_test
metrics-0.0.1-r2805: ERROR: ld.so: object 'libsandbox.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
metrics-0.0.1-r2805: chroot: /build/bob
metrics-0.0.1-r2805: cwd: /tmp/portage/chromeos-base/metrics-0.0.1-r2805/work/metrics-0.0.1/platform2/metrics
metrics-0.0.1-r2805: cmd: {/var/cache/portage/chromeos-base/metrics/out/Default/cumulative_metrics_test} '/var/cache/portage/chromeos-base/metrics/out/Default/cumulative_metrics_test'
metrics-0.0.1-r2805: [==========] Running 1 test from 1 test case.
metrics-0.0.1-r2805: [----------] Global test environment set-up.
metrics-0.0.1-r2805: [----------] 1 test from CumulativeMetricsTest
metrics-0.0.1-r2805: [ RUN      ] CumulativeMetricsTest.TestLoop
metrics-0.0.1-r2805: [0509/123036:ERROR:cumulative_metrics.cc(59)] cannot find CHROMEOS_RELEASE_VERSION
metrics-0.0.1-r2805: [0509/123036:WARNING:persistent_integer.cc(96)] cannot open /tmp/.org.chromium.Chromium.nFOaza/cumulative.testing.cycle.start for reading: No such file or directory
metrics-0.0.1-r2805: [0509/123036:WARNING:persistent_integer.cc(96)] cannot open /tmp/.org.chromium.Chromium.nFOaza/cumulative.testing.version.hash for reading: No such file or directory
metrics-0.0.1-r2805: [0509/123037:WARNING:persistent_integer.cc(96)] cannot open /tmp/.org.chromium.Chromium.nFOaza/cumulative.testing.x.pi for reading: No such file or directory
metrics-0.0.1-r2805: [0509/123037:WARNING:persistent_integer.cc(96)] cannot open /tmp/.org.chromium.Chromium.nFOaza/cumulative.testing.y.pi for reading: No such file or directory
metrics-0.0.1-r2805: ../../../../../../../tmp/portage/chromeos-base/metrics-0.0.1-r2805/work/metrics-0.0.1/platform2/metrics/cumulative_metrics_test.cc:67: Failure
metrics-0.0.1-r2805:       Expected: cm.Get(kMetricNameX)
metrics-0.0.1-r2805:       Which is: 100
metrics-0.0.1-r2805: To be equal to: 0
metrics-0.0.1-r2805: terminating with uncaught exception of type testing::internal::GoogleTestFailureException: ../../../../../../../tmp/portage/chromeos-base/metrics-0.0.1-r2805/work/metrics-0.0.1/platform2/metrics/cumulative_metrics_test.cc:67: Failure
metrics-0.0.1-r2805:       Expected: cm.Get(kMetricNameX)
metrics-0.0.1-r2805:       Which is: 100
metrics-0.0.1-r2805: To be equal to: 0
metrics-0.0.1-r2805: qemu: uncaught target signal 6 (Aborted) - core dumped
metrics-0.0.1-r2805: Error: /var/cache/portage/chromeos-base/metrics/out/Default/cumulative_metrics_test: failed with signal SIGIOT|SIGABRT(6)
metrics-0.0.1-r2805:  * ERROR: chromeos-base/metrics-0.0.1-r2805::chromiumos failed (test phase):
metrics-0.0.1-r2805:  *   (no error message)
metrics-0.0.1-r2805:  * 
metrics-0.0.1-r2805:  * Call stack:
metrics-0.0.1-r2805:  *     ebuild.sh, line  133:  Called src_test
metrics-0.0.1-r2805:  *   environment, line 3886:  Called platform_src_test
metrics-0.0.1-r2805:  *   environment, line 3454:  Called platform_pkg_test
metrics-0.0.1-r2805:  *   environment, line 3434:  Called platform_test 'run' '/build/bob/var/cache/portage/chromeos-base/metrics/out/Default/cumulative_metrics_test'
metrics-0.0.1-r2805:  *   environment, line 3487:  Called die
metrics-0.0.1-r2805:  * The specific snippet of code:
metrics-0.0.1-r2805:  *       "${cmd[@]}" || die

Luigi, what's the expectation of cumulative_metrics.cc's environment it is ran in?

Not a lot of expectations---pretty plain UNIX.
I am taking a look right now.

This is the output from running with FEATURES=test emerge-cyan metrics:


cmd: {/var/cache/portage/chromeos-base/metrics/out/Default/cumulative_metrics_test} '/var/cache/portage/chromeos-base/metrics/out/Default/cumulative_metrics_test'
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from CumulativeMetricsTest
[ RUN      ] CumulativeMetricsTest.TestLoop
[0509/195513:ERROR:cumulative_metrics.cc(59)] cannot find CHROMEOS_RELEASE_VERSION
[0509/195513:WARNING:persistent_integer.cc(96)] cannot open /tmp/.org.chromium.Chromium.9cmoAX/cumulative.testing.cycle.start for reading: No such file or directory
[0509/195513:WARNING:persistent_integer.cc(96)] cannot open /tmp/.org.chromium.Chromium.9cmoAX/cumulative.testing.version.hash for reading: No such file or directory
[0509/195513:WARNING:persistent_integer.cc(96)] cannot open /tmp/.org.chromium.Chromium.9cmoAX/cumulative.testing.x.pi for reading: No such file or directory
[0509/195513:WARNING:persistent_integer.cc(96)] cannot open /tmp/.org.chromium.Chromium.9cmoAX/cumulative.testing.y.pi for reading: No such file or directory
[       OK ] CumulativeMetricsTest.TestLoop (592 ms)
[----------] 1 test from CumulativeMetricsTest (592 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (592 ms total)
[  PASSED  ] 1 test.

Bob is aarch64, but the unit test is built and run on x86_64 so that should not matter.

There might be flakiness in the test.  I'll look for it.  Did it fail on any other platforms?

Yes, a lot of canary builds are affected, e.g. elm, fizz, gandof, grunt,...

So we're running the tests on the builders proper? I guess we just do that for unit tests in general -- even within a specific build target? 

#6 yes that's correct.  I guess some tests may change depending on the build, otherwise it might make more sense to just build and run them once.

I found the problem.  I think I'll have to revert that change for now.

https://chromium-review.googlesource.com/c/chromiumos/platform2/+/1012196

(also the CQ-DEPEND)

It's a timing issue.  I made incorrect assumptions on the semantics of the event loop.  I'll need to make the test more robust.

Revert CLs:

https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/1052968

https://chromium-review.googlesource.com/c/chromiumos/platform2/+/1052829

We chumped those reverts, but a build failed in an unexpected way:

https://luci-logdog.appspot.com/v/?s=chromeos/buildbucket/cr-buildbucket.appspot.com/8946971071489242832/+/steps/InitSDK/0/stdout

metrics-0.0.1-r2806: OSError: [Errno 2] No such file or directory: 'cumulative_metrics.h'

Do we need to manually bump the ebuild version?

bump it to r2807 ?

Yes, I think #10 is right.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/48d3f1b7ccf720a25c7dee65c1a7fcc0086e3439

commit 48d3f1b7ccf720a25c7dee65c1a7fcc0086e3439
Author: Luigi Semenzato <semenzato@chromium.org>
Date: Wed May 09 22:47:46 2018

metrics: manually bump ebuild version to ensure revert is effective.

BUG= https://crbug.com/841485 

Change-Id: I00e2496e6fa2ea3865d297f84e662d8e130a3ade
Reviewed-on: https://chromium-review.googlesource.com/1053241
Reviewed-by: Aaron Durbin <adurbin@chromium.org>
Reviewed-by: Benjamin Gordon <bmgordon@chromium.org>
Reviewed-by: Lann Martin <lannm@chromium.org>
Commit-Queue: Luigi Semenzato <semenzato@chromium.org>
Tested-by: Luigi Semenzato <semenzato@chromium.org>

[add] https://crrev.com/48d3f1b7ccf720a25c7dee65c1a7fcc0086e3439/chromeos-base/metrics/metrics-0.0.1-r2807.ebuild

Fixed a while ago.