i'm thinking this can only come up on incremental builders.  the scenario:
- unittest stage runs
- packages get rebuilt & installed with USE=test (via FEATURES=test)
- this includes trunks and chaps and cryptohome and such
- next run still have those packages built & installed w/USE=test enabled
- trunks gets upgraded but chaps does not
- we hit the failure situation you describe here

i think we didn't notice this previously because trunks & chaps were in the same git repo, and whenever one uprevved, the other was (more or less) guaranteed to be uprevved too.  now with the subtree work, that's no longer the case.

we could workaround it by having trunks & chaps list each other in their subtree vars ... that way we continue to guarantee they both get rebuilt.

i'm not sure if there's an emerge option atm that'd let us tell emerge it should autorebuild based on changed USE flags when the packages in question haven't been selected.  maybe -DN ?

It showed up on a variety of CQ builders.

https://uberchromegw.corp.google.com/i/chromeos/builders/master-paladin/builds/17797

> i think we didn't notice this previously because trunks & chaps were in the same git repo, and whenever one uprevved, the other was (more or less) guaranteed to be uprevved too.  now with the subtree work, that's no longer the case.

I also guessed this might be an unintended side effect from CROS_WORKON_SUBTREE, but what I don't quite understand is that, in the above log, both trunks and chaps were uprev'ed.


> i'm not sure if there's an emerge option atm that'd let us tell emerge it should autorebuild based on changed USE flags when the packages in question haven't been selected.  maybe -DN ?

I'm afraid it may now work; manpage of emerge(1) has following explanation about --newuse (-N):

 NOTE: This option ignores the state of the "test" USE flag, since that flag has a special binding to FEATURES="test"


===

Anyway, we need a short-term workaround for this issue. How about dropping [test] from those RDEPEND? I'm hesitant to add more dirs to CROS_WORKON_SUBTREE when they are not real dependencies...

I created https://chromium-review.googlesource.com/c/chromiumos/overlays/chromiumos-overlay/+/934107, but I'm not confident if it resolves the issue.

This breaks the CQ often enough that it should be at least P1.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/c3a66566c323a34eced508be6556deb9c526061f

commit c3a66566c323a34eced508be6556deb9c526061f
Author: Shuhei Takahashi <nya@chromium.org>
Date: Mon Feb 26 12:07:02 2018

Avoid runtime USE=test dependencies to trunks.

The purpose of these USE=test dependencies to trunks is to link
libtrunks_test.a, so they need not to be in RDEPEND.

I have not figured out the mechanism of  crbug.com/813791  yet, but
those runtime dependency seem to refuse trunks to be re-emerged
without USE=test.

Also updates EAPI to 5.

BUG= chromium:813791 
TEST=precq

Change-Id: Ie5a5875593370104f31560092603481fcc430383
Reviewed-on: https://chromium-review.googlesource.com/934107
Commit-Ready: Shuhei Takahashi <nya@chromium.org>
Tested-by: Shuhei Takahashi <nya@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>

[modify] https://crrev.com/c3a66566c323a34eced508be6556deb9c526061f/chromeos-base/tpm_manager/tpm_manager-9999.ebuild
[modify] https://crrev.com/c3a66566c323a34eced508be6556deb9c526061f/chromeos-base/chaps/chaps-9999.ebuild
[modify] https://crrev.com/c3a66566c323a34eced508be6556deb9c526061f/chromeos-base/attestation/attestation-9999.ebuild
[modify] https://crrev.com/c3a66566c323a34eced508be6556deb9c526061f/chromeos-base/cryptohome/cryptohome-9999.ebuild

I hope the issue is fixed. Please feel free to reopen if we still see the same failure.

I see a weird unit test failure with https://chromium-review.googlesource.com/c/chromiumos/platform2/+/857750#message-0aa628d3d1e81ac181d80de321c3d0b94c3c2887

Could it be related to this bug?

Yes, it looks possibly related... let me take a look.

I looked at the failing test in #c9 and I guess it's real flaky failure in the newly added tests. I'm closing the bug now, but please feel free to reopen if that's not the case.