To clarify further, this CL had a build time error that seems to have hit nearly every single board in the CQ, but passed the PreCQ.

PRECQ Samus (which passed):

https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/compile-only-pre-cq/builds/41897/steps/Uprev/logs/stdio

21:26:49: INFO: RunCommand: /b/cbuild/internal_master/chromite/bin/cros_mark_as_stable --all '--boards=samus' '--overlays=<snop>' '--drop_file=/b/cbuild/internal_master/src/scripts/cbuildbot_package.list' commit in /b/cbuild/internal_master
21:26:55: WARNING: Missing stable ebuild in /b/cbuild/internal_master/src/third_party/chromiumos-overlay/media-libs/arc-mesa
21:27:18: INFO: Committing changes with commit message: Marking 9999 ebuild for chromeos-base/plaso-cronlite as stable.
21:27:50: INFO: Committing changes with commit message: Marking 9999 ebuild for media-libs/arc-mesa as stable.
21:28:07: INFO: Running cidb query on pid 27322, repr(query) starts with <sqlalchemy.sql.expression.Update object at 0x7f7f7a568cd0>
21:28:11: INFO: cidb query succeeded after 1 retries


CQ Samus (which failed):

https://uberchromegw.corp.google.com/i/chromeos/builders/samus-paladin/builds/10593/steps/Uprev/logs/stdio

22:43:46: INFO: RunCommand: /b/cbuild/internal_master/chromite/bin/cros_mark_as_stable --all '--boards=samus' '--overlays=<snip>' '--drop_file=/b/cbuild/internal_master/src/scripts/cbuildbot_package.list' commit
cwd=/b/cbuild/internal_master
22:44:17: INFO: Attempting refresh to obtain initial access_token
22:44:17: INFO: Refreshing access_token
22:44:33: INFO: Committing changes with commit message: Marking 9999 ebuild for chromeos-base/plaso-cronlite as stable.
22:44:44: INFO: Committing changes with commit message: Marking 9999 ebuild for sys-kernel/lakitu-kernel-4_4 as stable.
22:45:48: ERROR: Package arc-mesa has a chromeos-version.sh script but it returned no valid version for "/b/cbuild/internal_master/src/third_party/arc-mesa"

I don't understand the difference between the uprev processes in the two environments.

off hand, i expect they both should have failed.  my guess is that the local repo sync state was diff, but i don't have a great reason for that position.

we should update the uprev message to include the version string that it actually marked as stable.  e.g. it should say:
22:44:33: INFO: Committing changes with commit message: Marking 9999 ebuild for chromeos-base/plaso-cronlite as stable as 0.0.1-r1234.

and when it fails, we should be logging the stdout & stderr.  there might be something interesting in there.  e.g. it should say:
22:45:48: ERROR: Package arc-mesa has a chromeos-version.sh script but it returned no valid version for "/b/cbuild/internal_master/src/third_party/arc-mesa"
22:45:48: ERROR: stdout: ...
22:45:48: ERROR: stderr: ...

I'll tweak the logging when I get a chance.