It looks to me like the builder is pulling in incorrect Chrome source code.

1) The non-compiling chrome code does not even exist in M50 branch. For whatever reason it looks like this build is pulling ToT chrome source, rather than from the M50 branch.



Evidence:
Given error: "platform/drm/gpu/gbm_buffer.cc:128:52: error: 'gbm_bo_get_format_modifier' was not declared in this scope"
 
Chrome only started calling that method last month: 
https://codereview.chromium.org/2039813002 ( - this CL should not be and is not in M50. Also I manually checked out refs/tags/50.0.2661.107 locally and grepped source and logs to confirm this)

The underlying platform API was only added to Chrome OS in May (long after M50 branch date): 
https://chromium-review.googlesource.com/346365
And that change never went to R50 CrOS branch:
https://chromium.googlesource.com/chromiumos/platform/minigbm/+log/release-R50-7978.B/gbm.h


2) Possible clue is in 
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/4972/steps/BuildPackages%20%5Bafdo_use%5D/logs/stdio

chromeos-chrome-50.0.2661.107_rc-r1:  * Using CHROME_VERSION = 50.0.2661.107
chromeos-chrome-50.0.2661.107_rc-r1:  * Skipping syncing as cbuildbot ran SyncChrome for us.
 

How can I verify cbuildbot actually ran SyncChrome for us?

@dshi (Infras Deputy) can you triage or investigate why the builds mentioned in OP appear to be using incorrect Chromium source code?

Interesting, so it looks like if it depends on SyncChrome, that does not appear to be happening here, at least I don't see the step in https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/4972/

That could explain it, I am not intimately familiar with how the chrome sync and build functions though.

Ahh, if you look at the chromeos_config.py on the branch, it says veyron_rialto does not use Chrome, so it is configured to not do any Chrome steps... yet Chrome is still in the dependency graph for the build.

...
  site_config.Add(
      'veyron_rialto-release', _release,
      _base_configs['veyron_rialto'],
      # rialto does not use Chrome.
      sync_chrome=False,
      chrome_sdk=False,
  )
...

Good find! 

maybe just a cherry pick of  https://chromium-review.googlesource.com/343623 then?

Not sure why this regressed in the release branch. Is chromite not version controlled with the rest of the source?

The following revision refers to this bug:
  https://chromium.googlesource.com/chromiumos/chromite/+/c2ea2cff4ffbacec68f8640c9dc20f26262e92b3

commit c2ea2cff4ffbacec68f8640c9dc20f26262e92b3
Author: Bernie Thompson <bhthompson@google.com>
Date: Fri Jul 22 00:14:40 2016

Make veyron_rialto build like other veyron boards

It seems veyron_rialto does use Chrome, so we should have the build steps
such that it can build properly.

BUG= chromium:630457 
TEST=None

Change-Id: I4fcbe57cb6a5d808c48398c5b6a2f6ff519b20b8
Reviewed-on: https://chromium-review.googlesource.com/362481
Reviewed-by: Jonathan Dixon <joth@chromium.org>
Reviewed-by: Bernie Thompson <bhthompson@chromium.org>
Commit-Queue: Bernie Thompson <bhthompson@chromium.org>
Tested-by: Bernie Thompson <bhthompson@chromium.org>

[modify] https://crrev.com/c2ea2cff4ffbacec68f8640c9dc20f26262e92b3/cbuildbot/config_dump.json
[modify] https://crrev.com/c2ea2cff4ffbacec68f8640c9dc20f26262e92b3/cbuildbot/chromeos_config.py

When these were built on the waterfall the builds were on a shared builder using a group, which is why we get some other boards output when you build veyron_rialto as well. 

I suspect those other boards would be effectively updating the Chrome sources for us in that case. When we tried to build a one off with just veyron_rialto with the manual build it seems to end up building whatever Chrome happens to be checked out on the machine that gets allocated for the build.

The builder is no longer failing with this error. Here is the last good build:
https://uberchromegw.corp.google.com/i/chromiumos.tryserver/builders/release/builds/5060

bulk verified